An ESXi upgrade initiated from the SDDC UI does not happen in parallel but proceeds sequentially

Products

• VMware SDDC Manager

Issue/Introduction

An ESXi upgrade initiated from the SDDC does not happen in parallel despite the fact that the Sequential Upgrade option has NOT been selected in the upgrade wizard.

lcm logs:

YYYY-MM-DDTHH:MM:SS.XXXX INFO  [vcf_lcm,xxxxxxxxxxxxxxxx,87b5] [c.v.v.l.r.a.c.v.u.UpgradeController,http-nio-127.0.0.1-7400-exec-10] Upgrade Specification UpgradeSpec(bundleId=00712db6-a489-4d50-b253-e53f7e13d726, resourceType=CLUSTE
R, parallelUpgrade=false, draftMode=false, resourceUpgradeSpecs=[ResourceUpgradeSpec(resourceId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx, shutdownVms=null, toVersion=null, scheduledTimestamp=null, upgradeNow=true, personalitySpec=null, custo
mIsoSpec=null, enableQuickboot=false, evacuateOfflineVms=true, esxUpgradeOptionsSpec=null, hostsToUpgrade=null), ResourceUpgradeSpec(resourceId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx, shutdownVms=null, toVersion=null, scheduledTimestamp=nu
ll, upgradeNow=true, personalitySpec=null, customIsoSpec=null, enableQuickboot=false, evacuateOfflineVms=true, esxUpgradeOptionsSpec=null, hostsToUpgrade=null)], nsxtUpgradeUserInputSpecs=null, vcenterUpgradeUserInputSpecs=null)

 

This can greatly increase the time required to complete the host upgrade in environments with a large amount ESXi host clusters.

Environment

VMware Cloud Foundation 5.2.1

Cause

VMware by Broadcom is aware of this issue and will include a fix in a future release.

Resolution

As a workaround initiate the upgrade via API rather than from the UI:

Example:

API Endpoint: https://<sddc-manager-fqdn>/v1/upgrades'

Payload:

{
    "bundleId": "<bundleId>",
    "resourceType": "CLUSTER",
    "parallelUpgrade": true,
    "resourceUpgradeSpecs": [
        {
            "resourceId": "<clusterId>",
            "upgradeNow": false,
            "enableQuickboot": true,
            "evacuateOfflineVms": true,
            "esxUpgradeOptionsSpec": {
                "enableQuickPatch": false
            },
            "personalitySpec": {
                "personalityId": "<personalityId>"
            }
        }
    ],
    "draftMode": true
}

Notes -

- if there are multiple clusters, additional coma separated values can be added to resourceUpgradeSpecs. 

- the bundleId can be found in the SDDC Manager UI > Lifecycle Management > Bundle Management

• select "View Details" for the relevant bundle
• Bundle ID is shown in the form xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

- the clusterId can be found using the SDDC Manager UI > Inventory > Workload Domains

• select the relevant domain, from the Clusters tab - select the relevant cluster
• look in the URL - the string after .../clusters/ of the form xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx is the clusterId

- the personalityID can be found using the SDDC Manager UI > Developer Center > API Explorer

• search for personalities
• this will be returned - > GET / v1/personalities
• execute it and information that includes the personalityId for the target ESX version is returned
• a given personalityId should be associable with the version of ESX that is being upgraded to

- the evacuateOfflineVms can be modified to false in cases where the VMs(eg, monitoring servers) need to go down with the ESXi host. 

 

Additional Information

If the upgrade is performed using custom image the parameters will alter as below:

{
    "bundleId": "<bundleId>",
    "resourceType": "CLUSTER",
    "parallelUpgrade": true,
    "resourceUpgradeSpecs": [
        {
            "resourceId": "<cluster1Id>",
            "esxUpgradeOptionsSpec": {
                "enableQuickboot": true,
                "evacuateOfflineVms": true,}
         },
        {
            "resourceId": "<cluster2Id>",
            "esxUpgradeOptionsSpec": {
                "enableQuickboot": true,
                "evacuateOfflineVms": true,}
        }
    ]
}

Note: Above example shows for 2 clusters can add more values in similar way if needed.