Add Cluster or Host fails with error 504 Gateway Time-out in VMware Cloud Foundation on Vxrail

Products

• VMware Cloud Foundation

Issue/Introduction

• Increase the timeout value for nginx and SDDC manager UI in the VCF on VxRail deployment prior to VCF version 3.9.1.

Symptoms:

• Adding a Cluster or Esxi host(s) fails with a 504 Gateway Time-out in the SDDC Manager UI.
• Attempting to log in to the SDDC Manager UI, reports :  Bad gateway, nginx/1.15.

Environment

VMware Cloud Foundation on VxRail 3.8.x
VMware Cloud Foundation 3.9.x
VMware Cloud Foundation on VxRail 3.7.x

Cause

• In large scale VMware Cloud Foundation on VxRail deployments, the discovery of the Cluster or ESXi hosts takes more than the default time-out value for the "nginx" and "SDDC Manager UI services.
  • Nginx  gateway  default timeout  value is 1 minute.
  • SDDC Manager UI server default timeout value is 2 minute.

Resolution

• This is a known issue affecting the mentioned versions, this will be fixed in future release. Please see the below section to workaround this issue.

Workaround:

1. Take a snapshot of SDDC manager Controller VM.
2. Download the extract  vxrail-scale-fixitup-v2.2.py script attached to the KB.
3. Use a file transfer utility to upload the vxrail-scale-fixitup-v2.2.py script to the /tmp/ directory on the SDDC Manager Controller VM.
4. SSH to the SDDC Manager Controller VM as the vcf user.
5. Issue the following command to switch to the root user account:

# su

6. Run the following command to check the health state of nginx.

# nginx -t

 

Example:

 

root@sddc-manager [ ~ ]# nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Note: Do not run the vxrail-scale-fixitup-v2.2.py script if nginx.conf  is not in a healthy state.
 

7. Run the following command to confirm the sddc-manager-ui-app service is up.

# curl localhost/sddc-manager-ui-app/about | json_pp

Example:
root@sddc-manager [ ~ ]# curl localhost/sddc-manager-ui-app/about | json_pp
{
   "author" : "VMware",
   "version" : "3.8.0-RELEASE-14212073",
   "dbSchemaVersion" : 1,
   "builtBy" : "mts",
   "createdBy" : "Apache Maven 3.5.0",
   "buildDate" : "2019-07-22T03:13:24Z",
   "id" : "c091bdb5-6558-40e6-9011-2794e9fc61de",
   "name" : "SDDC-MANAGER-UI-APP"

8. Navigate to path /tmp/ directory.
9. Run the following command to execute the script to increase the nginx and SDDC_MANAGER_UI time out.

# python vxrail-scale-fixitup-v2.2.py

 

Note: The script will backup the following files at their original locations before modifying them:

-  /opt/vmware/vcf/sddc-manager-ui-app/server/src/servers/sddcManagerServer/www.js file.
- /etc/nginx/nginx.conf file.

10. Re-run the commands in steps 6 and 7 to verify the status of nginx and sddc-manager-ui-app  after running the vxrail-scale-fixitup-v2.2.py script.

 

Note:

• If nginx  and sddc-manager-ui-app are not in healthy state after running the script and and the SDDC Manager UI is not loading, please restore the www.js and  the nginx.conf files from their backups or revert the SDDC Manager to the snapshot that was taken before running the script. 
• The  vxrail-scale-fixitup-v2.2.py script should only be run once on a system, running  it twice  without restoring  www.js and nginx.conf files from their backup files or reverting the snapshot will cause nginx  service to not startup.
• The VCF upgrade to any version prior to VCF 3.9.1 will overwrite the www.js and  nginx.conf files and revert the Time-out value back to default. So the vxrail-scale-fixitup-v2.2.py script will need to be re-run again after upgrading to any VCF version prior to 3.9.1.