Error 'Http failure response for <URL>: 504 OK' seen on VCF Operations UI while performing upgrade/patch

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

In VMware Cloud Foundation (VCF) environments, accessing the SDDC Manager UI or VCF Operations interface results in repeated REST API failures and 504 gateway timeout errors.

Specific symptoms include:

• VCF Operations UI reports failure responses for SDDC manager:

  • Http failure response for https://<VCF_Operations_FQDN>/vcf-operations/plug/sddc-##/plugin/api/v1/releases?includeOnlyCompatible=true: 504 OK

  • Http failure response for https://<VCF_Operations_FQDN>/vcf-operations/plug/sddc-##/plugin/api/v1/services-config: 504 OK

  • Error while fetching storage information : Http failure response for https://<VCF_Operations_FQDN>/vcf-operations/plug/sddc-##/plugin/api/v1/system/bundles-directory/storage-info: 504 OK

• SDDC Manager UI 'Dashboard' displays the following error:

  • Failed to fetch lcm info: Http failure response for https://<SDDC_FQDN>/ui/api/internal/lcm/domains/completed: 504 OK

• /var/log/vmware/vcf/vcf-commonsvcs/vcf-commonsvcs.log shows that all the LCM APIs fail:
  • org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost/lcm/about": Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost/lcm/internal/upgrades/tasks/#####################": Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost/lcm/internal/umds/tasks/sync/#####################": Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost/lcm/internal/upgrades/tasks/#####################": Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost/lcm/internal/vsan-hcl/tasks/sync/#####################": Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost/lcm/internal/vsan-hcl/tasks/sync/#####################": Read timed out

Environment

 

• SDDC Manager 9.x

• VCF Operations 9.x

 

Cause

The local lcm service daemon on the SDDC Manager appliance becomes unresponsive or hung, causing commonsvcs to experience read timeouts.

Resolution

Restart the Lifecycle Management (lcm) service on the SDDC Manager appliance:

1. Open an SSH session to the SDDC Manager appliance using the vcf user credentials, then switch to the root user:

   su -
   

2. Restart the lcm service daemon:

   systemctl restart lcm
   

3. Verify that the service has returned to an active/running state:

   systemctl status lcm