SDDC Manager LCM can incur up to 3-hour additional delay compared to the ESXi upgrade duration in some network-constrained environment
search cancel

SDDC Manager LCM can incur up to 3-hour additional delay compared to the ESXi upgrade duration in some network-constrained environment

book

Article ID: 388721

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • SDDC Manager LCM orchestrator can wait 80-90 min for the upgrade schedule to start or upgrade completion to finish if access to storage.googleapis.com is blocked
  •  The delay can also happen if there is an ongoing scheduled bundle downloads.
  • The lcm-debug log shows below entries
    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,689e46c5611ebcf5a7cf96356875dfe5,bb66] [c.v.e.s.l.b.d.depot.DepotDownloader,vac-scheduler-1] Connection closed
    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,689e47670f2f5c743a3873179cba4d04,9891] [c.v.e.s.l.c.s.VmwareCompatibilityDataDownloadScheduler,vac-scheduler-1] Compatibility Metadata Download Scheduler
    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,689e47670f2f5c743a3873179cba4d04,ab3e] [c.v.e.s.l.c.s.VmwareCompatibilityDataDownloadScheduler,vac-scheduler-1] Starting VVS Compatibility Data download
    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,689e47670f2f5c743a3873179cba4d04,ab3e] [c.v.e.s.l.s.i.DepotConfigServiceImpl,vac-scheduler-1] Offline depot enabled status: false
    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,689e47670f2f5c743a3873179cba4d04,ab3e] [c.v.v.c.c.i.v.r.client.VvsApiClient,vac-scheduler-1] Getting entire set of product bundle interop/upgrade details.
    YYYY-MM-DDTHH:MM:SS INFO  [vcf_lcm,689e47670f2f5c743a3873179cba4d04,ab3e] [c.v.v.c.c.i.v.r.client.VvsApiClient,vac-scheduler-1] vvs uri with query params: https://vvs.esp.vmware.com/v1/products/bundles/type/vcf-lcm-v2-bundle?format=json
    YYYY-MM-DDTHH:MM:SS INFO  [vcf_lcm,689e47670f2f5c743a3873179cba4d04,ab3e] [o.a.h.c.h.i.c.HttpRequestRetryExec,vac-scheduler-1] Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://storage.googleapis.com:443

 

Environment

  • SDDC Manager 5.2
  • SDDC Manager 5.2.1
  • SDDC Manager 5.2.1.1

 

Cause

The shared pool of LCM scheduled tasks got packed into a single thread. "vac-scheduler-1" frequently appears in the SDDC LCM log. 

Both precheck and upgrade tasks are not impacted. However, the orchestration scheduler tasks are impacted.

Resolution

This issue is resolved in SDDC Manager 5.2.1.2 and 9.0 versions.

To workaround apply one of the below.

Workaround A:


Workaround B:

  • Download bundles before starting upgrades


Workaround C:

Note: This is a one-time-only change. 

  1. SSH to SSDC Manager and switch to root.
  2. Go to the home/vcf dir:
    cd /home/vcf

  3.  Create this file:  telemetry-vac.properties
    touch telemetry-vac.properties

  4. Add the following line: vac.scheduler.corepoolsize=10
    echo ""vac.scheduler.corepoolsize=10"" > telemetry-vac.properties

  5. Update permissions and ownership on the file:
    chown vcf_lcm telemetry-vac.properties
    chgrp vcf telemetry-vac.properties

  6. Restart LCM:
    systemctl restart lcm

Additional Information

VMware Cloud Foundation 5.2.1.2 Release Information

Upgrading from 5.2.1.1
SDDC Manager is the only component that requires an upgrade. See Independent SDDC Manager Upgrade using the SDDC Manager UI.
Upgrading from 5.2.1
You cannot upgrade directly from 5.2.1 to 5.2.1.2. Use the following process instead:
    Upgrading from 5.2:
    See Flexible BOM Upgrade in VMware Cloud Foundation. When selecting the target version for SDDC Manager choose the version listed in the BOM update table below.
    In order to upgrade from 5.2 to 5.2.1.2, you must download the bundles for both SDDC Manager 5.2.1.0 and SDDC Manager 5.2.1.2.

    Resolved Issues:
    • LCM scheduled tasks are running in a single thread causing bundle download delay and failures.
      Powered by Wolken Software