VMware Cloud Foundation navigating to Update/Patches tab gives error "Retrieving all scheduled and in progress bundles failed. Http failure response" 504 Gateway Time-out
search cancel

VMware Cloud Foundation navigating to Update/Patches tab gives error "Retrieving all scheduled and in progress bundles failed. Http failure response" 504 Gateway Time-out

book

Article ID: 313835

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

This document will list the steps to apply the HotPatch to fix the above issue.

Symptoms:
  • This issue is seen in a scaled setup.
  • From the SDDC Manager UI, on navigating to Update/Patches tab the error "Retrieving all scheduled and in-progress bundles failed. Http failure response" 504 Gateway Time-out is seen.
  • Following LCM APIs are taking time (sometimes more than one minute around 1.45 minutes to 2 minutes). The API response time worsens when called concurrently.

   - /v1/upgrades

   - /lcm/inventory/upgrades


Environment

VMware Cloud Foundation 3.10.2.1

Cause

In a scaled setup the API calls time out.

Resolution

Hotpatch attached to the knowledge base article has been provided to address the issue.

Please follow the below steps to apply the hotpatch:
  • Take Snapshot of SDDC Manager (by logging Management VC)
  • Download the vcf31021-hp-JULY-2021.tar file to /home/vcf
  • Login to SDDC Manager VM using vcf user and change to "root" user
  • cd /home/vcf/
  • Extract the vcf31021-hp-JULY-2021.tar file (tar -xvf vcf31021-hp-JULY-2021.tar)
  • Back up the current binaries (jars)

root@sddc-manager [ /home/vcf ]# mkdir backup
root@sddc-manager [ /home/vcf ]# cd backup/

LCM jar back up:
  1. Stop the lcm service.
systemctl stop lcm
  1. Take a back up of current lcm binaries (Jars)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-adapters-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-app-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-common-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-orch-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-services-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar$(date +%Y%m%d)
cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)

SDDC Manager UI back up: 
  1. Stop the sddc-manager-ui-app service
systemctl stop sddc-manager-ui-app
  1. Take a back up of current UI files
cp -r /opt/vmware/vcf/sddc-manager-ui-app/ /home/vcf/backup/sddc-manager-ui-app.$(date +%Y%m%d)/
 
 
  • Verify the backed up LCM  jars and SDDC Manager UI files at /home/vcf/backup 
Replace the new LCM jars and SDDC Manager UI RPM and start the service: 
  1. cd /home/vcf/vcf31021-hp-JULY-2021 
  2. Replace lcm jars
cp lcm-adapters-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jar
cp lcm-app-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jar
cp lcm-common-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jar
cp lcm-orch-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jar
cp lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar
cp lcm-services-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jar
cp lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar
cp lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar
cp caffeine-2.7.0.jar /opt/vmware/vcf/lcm/lcm-app/lib/caffeine-2.7.0.jar
cp spring-context-support-5.2.3.RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/spring-context-support-5.2.3.RELEASE.jar
  1. Make sure replaced lcm jars has below permission and ownership.

cd /opt/vmware/vcf/lcm/lcm-app/lib

chown vcf_lcm:vcf caffeine-2.7.0.jar
chown vcf_lcm:vcf  spring-context-support-5.2.3.RELEASE.jar

chmod 750 caffeine-2.7.0.jar
chmod 750  spring-context-support-5.2.3.RELEASE.jar

-rwxr-x--- 1 vcf_lcm vcf lcm-adapters-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf lcm-app-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf lcm-common-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf lcm-orch-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf lcm-services-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf caffeine-2.7.0.jar
-rwxr-x--- 1 vcf_lcm vcf spring-context-support-5.2.3.RELEASE.jar

Example Commands :

# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 191004 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 85067 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 57424 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 56967 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 167907 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 352472 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 96078 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar
#ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 744570 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/caffeine-2.7.0.jar
-rwxr-x--- 1 vcf_lcm vcf 846706 Jul 2 10:34 /opt/vmware/vcf/lcm/lcm-app/lib/caffeine-2.7.0.jar
# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/spring-context-support-5.2.3.RELEASE.jar
-rwxr-x--- 1 vcf_lcm vcf 182130 Jul 2 10:52 /opt/vmware/vcf/lcm/lcm-app/lib/spring-context-support-5.2.3.RELEASE.jar

  1. Update the lcm properties files.
  • Take backup of /opt/vmware/vcf/lcm/lc-app/conf/application-prod.properties and /opt/vmware/vcf/lcm/lcm-app/conf/jpa-prod.properties

cp -fp /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties /home/vcf/backup/application-prod.properties.$(date +%Y%m%d)

cp -fp /opt/vmware/vcf/lcm/lcm-app/conf/jpa-prod.properties /home/vcf/backup/jpa-prod.properties.$(date +%Y%m%d)

  • Add the below content to /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties file.

lcm.inventory.upgrade.cache.initial.capacity=2
lcm.inventory.upgrade.cache.maxsize=4
lcm.inventory.upgrade.cache.expiry.seconds=300

  • Add the below content to /opt/vmware/vcf/lcm/lcm-app/conf/jpa-prod.properties filelcm.datasource.socket.timeout=30
  1. Upgrade the SDDC Manager UI RPM:
    • cd /home/vcf/vcf31021-hp-JULY-2021
  • rpm -Uvh sddc-manager-ui-app-3.10.2-18290131.noarch.rpm​​​​
  1. Start the LCM service.
#systemctl start lcm
  • Please Wait for two minutes for service to come up. 

  • You can check the status using 

# curl localhost/lcm/about | json_pp
  

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 212 0 212 0 0 4510 0 --:--:-- --:--:-- --:--:-- 4608
{
"buildJdk" : "1.8.0_221",
"serviceId" : "defaultUUID",
"name" : "LCM",
"buildDate" : "2021-05-11 15:21:24 UTC",
"author" : "VMware",
"builtBy" : "mts",
"createdBy" : "Apache Maven 3.5.0",
"version" : "3.10.2-vcf31021RELEASE-18015031"

}
  1. Start the SDDC manager UI service:
    1. systemctl restart sddc-manager-ui-db.service 
(wait for 1 minute)
  1. systemctl start sddc-manager-ui-app 
(wait for two minutes for the service to come up)

You can check the status using 

$ curl localhost/sddc-manager-ui-app/about | json_pp 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   240  100   240    0     0  48000      0 --:--:-- --:--:-- --:--:-- 48000
{
   "builtBy" : "mts",
   "dbSchemaVersion" : 1,
   "createdBy" : "Apache Maven 3.5.0",
   "author" : "VMware",
   "name" : "SDDC-MANAGER-UI-APP",
   "version" : "3.10.2-vcf31021RELEASE-18290131",
   "id" : "4a0a3a15-496f-4aa1-9040-d11c3eb8df43",
   "buildDate" : "2021-07-08T12:48:46Z"
}
  1. Apply the VersionAlias for SDDC_MANAGER_UI (as version of the SDDC Manager UI is changed)
  • Take backup of /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml file.
cp -fp /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml /home/vcf/backup/VersionAlias.yml.$(date +%Y%m%d)
  • Run the following curl command to add SDDC_MANAGER_UI specific version alias information.

curl 'http://localhost/v1/system/settings/version-aliases/SDDC_MANAGER_UI/3.10.2-vcf31021RELEASE-18015030' -X PUT -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"aliases" : [ "3.10.2-vcf31021RELEASE-18290131" ], "forceUpdate" : true}'​​​​​​​​

  • Verify that the below content is added /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml file.

    SDDC_MANAGER_UI:
            - base: 3.10.2-vcf31021RELEASE-18015030
              alias:

                  - 3.10.2-vcf31021RELEASE-1829013

NOTE:

  1. Having Multiple SDDC Manager UI sessions usage might cause delay in UI response in larger VCF deployments. Recommendation is to have one or two SDDC Manager UI sessions.
  2. Also the following nginx configuration needs to be done( if not done already)

  • Take a backup of /etc/nginx/nginx.conf

  • Add/Modify the below routes in http section.
       
    location  ~ ^/v1/(bundles|upgradables|upgrades|system)(.*) {
                proxy_read_timeout   600;           
                proxy_connect_timeout   600;        
                proxy_send_timeout   600;           
                proxy_pass http://127.0.0.1:7400/v1/$1$2$is_args$args;
            }
            location /lcm/ {
                proxy_read_timeout   600;          
                proxy_connect_timeout   600;      
                proxy_send_timeout   600;          
                proxy_pass http://127.0.0.1:7400;
            }
            #adding timeout for lcm internal api after /lcm/ route
            location ~ ^/internal/lcm/(.*) {            
                proxy_read_timeout   600;               
                proxy_connect_timeout   600;            
                proxy_send_timeout   600;               
                proxy_pass http://127.0.0.1:7500;       
            } 

  • Add/Modify the below routes in https section

 # lcm v1 public APIs
        location  ~ ^/v1/(bundles|upgradables|upgrades|system)(.*) {
            proxy_read_timeout   600;                             
            proxy_connect_timeout   600;                          
            proxy_send_timeout   600;                             
            auth_basic           "closed site";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_pass http://127.0.0.1:7400/v1/$1$2$is_args$args;
        }
        #Added this after sddc-manager-ui route
        location ~ ^/internal/lcm/(.*) {   
            proxy_read_timeout   600;
            proxy_connect_timeout   600;
            proxy_send_timeout   600;
            auth_basic           "closed site";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_pass http://127.0.0.1:7500;

        }
  • Save the file.
  • Verify the syntax: nginx -t
  • Restart nginx: systemctl restart nginx


Attachments

vcf31021-hp-JULY-2021 get_app
Powered by Wolken Software