SDDC Manager LCM can incur up to 3-hour additional delay compared to the ESXi upgrade duration in some network-constrained environment

Products

• VMware SDDC Manager

Issue/Introduction

SDDC Manager LCM orchestrator can wait 80-90 min for the upgrade schedule to start or upgrade completion to finish if access to storage.googleapis.com is blocked

The delay can also happen if there is an ongoing scheduled bundle downloads.

Environment

SDDC Manager 5.2.x

Cause

The shared pool of LCM scheduled tasks got packed into a single thread. "vac-scheduler-1" frequently appears in the SDDC LCM log. 

Both precheck and upgrade tasks are not impacted. However, the orchestration scheduler tasks are impacted.

Resolution

Workaround A:

• Ensure URL storage.googleapis.com is not block within the environment.
• Reference: Public URL needed by SDDC Manager for online functionality:
  https://knowledge.broadcom.com/external/article/327186/public-url-list-for-sddc-manager.html 

Workaround B:

• Download bundles before starting upgrades

Workaround C:

Note: This is a one-time-only change. 

1. SSH to SSDC Manager and switch to root.
2. Go to the home/vcf dir:
   cd /home/vcf
   
   
3.  Create this file:  telemetry-vac.properties
   touch telemetry-vac.properties
   
   
4. Add the following line: vac.scheduler.corepoolsize=10
   echo ""vac.scheduler.corepoolsize=10"" > telemetry-vac.properties
   
   
5. Update permissions and ownership on the file:
   chown vcf_lcm telemetry-vac.properties
chgrp vcf telemetry-vac.properties
   
   
6. Restart LCM:
   systemctl restart lcm