Fetching bundles fails with 504 gateway timeout error in SDDC Manager

Products

• VMware Cloud Foundation

Issue/Introduction

To fasten the LCM API's to respond

Symptoms:
SDDC Manager Bundle Management shows the below error

Retrieving all applicable bundles failed. Http failure response for https://<VC_fqdn>/ui/api/internal/lcm/repository/true: 504 Gateway Time-out

lcm logs are overwhelmed with excessive debug log statements as below:

YYYY-MM-DDT19:11:29.487+0000 DEBUG [vcf_lcm,cf2dca2e824e4bb1,b8ed] [c.v.e.s.l.d.c.m.LcmManifestClientImpl,http-nio-127.0.0.1-7400-exec-15] Manifest Service model { "version": 1, "sequenceNumber": 2, "creationTime": 1603257914075, "publishedDate": 1605761200653, "releases": [ { "product": "VCF", "version": "4.0.0.0", "description": "\

 

Environment

VMware Cloud Foundation 4.1
VMware Cloud Foundation 4.x

Cause

The LCM get available upgrade API response happens to be very slow as a result fetching the bundles happen to time-out

Resolution

1. Take a snapshot of SDDC Manager VM
2. ssh to sddc manager with vcf user and su to root
3. Take a backup of logback-prod.xml file
   • cp /opt/vmware/vcf/lcm/lcm-app/conf/logback-prod.xml /opt/vmware/vcf/lcm/lcm-app/conf/logback-prod.xml.bkp
4. Disable the DEBUG level logging
   • vi /opt/vmware/vcf/lcm/lcm-app/conf/logback-prod.xml
   • for the highlighted entry in logback-prod.xml change it to "INFO" from "DEBUG" to reduce the logging
   • 
5. Restart lcm service
   • systemctl restart lcm
6. Increase the nginx timeout - Follow the steps below
   • chmod 0700 /etc/nginx/nginx.conf
   • Note: Take a backup of this file prior to proceeding further
   • vi /etc/nginx/nginx.conf
   • Under the http section, above the server section add the following lines:
   • proxy_read_timeout 600;
     proxy_connect_timeout 600;
     proxy_send_timeout 600;
   • below image is for its reference save and exit the file
   • 
7. Validate that the re-configured nginx file is valid
   • nginx -t
8. Restart the nginx service
   • systemctl restart nginx