Password/Credentials rotate operation in VMware SDDC Manager fails with 504 gateway timed out

Products

• VMware Cloud Foundation

Issue/Introduction

• SDDC Manager UI shows "504 gateway timed out" error.
• The workflows fails in a large scale environment or an old deployments with large amount of historical credentials/password rotate operations.
• Attempting to retry the workflow also shows "504 gateway timed out." 

Cause

The backend services take more than 60 seconds to fetch details of the FAILED task to provide RETRY option but since UI gateway at nginx is configured to wait only for 60 seconds so it sends back a timeout error message to the UI. 

Resolution

To fix the issue in existing environments it's advised to increase the gateway timeout from 60 seconds to 900 seconds. This will allow the SDDC Manager UI to wait longer to complete the operation.
In the unlikely event the timeout error is still observed after making the below changes, the timeout value can be increased to an even larger value, for example 1200 seconds.

Below is the workaround needed to be applied to nginx.conf file where we increase the timeout for credentials API's.
 

Steps to update nginx.conf file:

1. Open a SSH session to the SDDC Manager VM and switch to the root user:

su -

2. Using a text editior, open the nginx.conf file at following location /etc/nginx/nginx.conf.
3. Search for the string "credentials" where the line matches the following:

location  ~ ^/v1/(hosts|license-keys|credentials|certificate-authorities|skyline)":

4. Modify the line from the following:

# operationsmanager v1 public APIs

        location ~ ^/v1/(hosts|license-keys|credentials|certificate-authorities|skyline)(.*) {

            proxy_pass http://127.0.0.1:7300/operationsmanager/v1/$1$2$is_args$args;

        }
 

Replacing with:
 

   # operationsmanager v1 public APIs
        location ~ ^/v1/(hosts|license-keys|credentials|certificate-authorities|skyline)(.*) {
            proxy_read_timeout 900;
            proxy_connect_timeout 900;
            proxy_send_timeout 900;
            proxy_pass http://127.0.0.1:7300/operationsmanager/v1/$1$2$is_args$args;
        }

For requests coming from the UI, re-configure the nginx.conf file specifically for UI server request using the below steps:

1. Search for the string ""location ~ /ui/.* {"" where the line matches the following.
2. Add the entry to reflect the same as below:

location ^~ /ui/api/v1/credentials {
        proxy_read_timeout 900;
        proxy_connect_timeout 900;
        proxy_send_timeout 900;

# UI server handles auth via PSC/SSO
                proxy_pass http://127.0.0.1:7500;
            }

3. Restart the nginx servce:

systemctl restart nginx.service

The timeout setting on the nodejs server service will also need to be increased from the default of 2 minutes.

1. Open a SSH session to the SDDC Manager VM and switch to the root user:

su -

2. Stop SDDC Manager UI service

systemctl stop sddc-manager-ui-app

3. Update UI server connection timeout value by editing /opt/vmware/vcf/sddc-manager-ui-app/server/src/servers/sddcManagerServer/www.js file
4. Search for the string "server.listen" in the file and Add line "server.timeout = 900000; " below.
5. Save and exit the file.
6. Start SDDC Manager UI service 

systemctl start sddc-manager-ui-app

7. Allow some time of the service to start, check the status by running:

systemctl status sddc-manager-ui-app

8. At this time, the password workflow can be started.