Creating new Cluster through SDDC Manager fails at "Upload vSphere Lifecycle Manager Image to vCenter Server"

Products

VMware SDDC Manager

Issue/Introduction

The "Upload vSphere Lifecycle Manager Image to vCenter Server" sub-task fails during add cluster workflow with Failed to upload Image Zip message:

Failed to upload Image Zip [https://FQDN_OF_SDDC-Manager/vmware/vcf/personalities/xxxxxxxxxxxxxxxxxx/Content/OFFLINE_BUNDLE_xxxxx-xxxx-xxxxxx.zip] to vCenter.

Visible error in the vCenter Web Client:
```
Import Update Task stuck at 80% 
```

Checking Domain Manager service log for SDDC Manager(var/log/vmware/vcf/domainmanager/domainmanager.log) we can observe similar snippets:

YYYY-MM-DDTHH:MM:SS+0000 ERROR [vcf_dm,opid-of-task,8cc6] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-15]  [Reference Token of failed subtask from SDDC] VSPHERE_FAILED_TO_UPLOAD_ZIP_TO_VCENTER Failed to upload Image Zip [https://FQDN_OF_SDDC-Manager/vmware/vcf/personalities/xxxxxxxxxxxxxxxxxx/Content/OFFLINE_BUNDLE_xxxxx-xxxx-xxxxxx.zip] to vcenter.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to upload Image Zip [https://FQDN_OF_SDDC-Manager/vmware/vcf/personalities/xxxxxxxxxxxxxxxxxx/Content/OFFLINE_BUNDLE_xxxxx-xxxx-xxxxxx.zip] to vcenter.


Caused by: java.util.concurrent.TimeoutException: Task did not complete within 0 seconds.Current task state is Info (com.vmware.cis.task.info) => {
    progress = Progress (com.vmware.cis.task.progress) => {
        total = 100,
        completed = 80,
        message = LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
            id = com.vmware.vcIntegrity.lifecycle.Task.Progress,
            defaultMessage = Current progress for task created by VMware vSphere Lifecycle Manager,
            args = [],
            params = <null>,
            localized = Current progress for task created by VMware vSphere Lifecycle Manager
        }
    },
 
    status = RUNNING,
    cancelable = false,
    error = <null>,
    startTime = YYYY-MM-DDTHH:MM:SS.Z[GMT],
    endTime = <null>,
    user = [email protected]
    [dynamic fields]: {
        last_update_time = YYYY-MM-DDTHH:MM:SS.Z,
        subtasks = <unset>,
        notifications = com.vmware.esx.settings.notifications => {warnings=[], errors=[], info=[]}
    }
}
        at com.vmware.vcf.vapi.vsphere.PersonalityImageManager.monitorTask(PersonalityImageManager.java:963)
        at com.vmware.vcf.vapi.vsphere.PersonalityImageManager.pullImageToOfflineDepotTask(PersonalityImageManager.java:1082)
        at com.vmware.vcf.vapi.vsphere.PersonalityImageManager.pullImageToOfflineDepot(PersonalityImageManager.java:1042)
        at com.vmware.vcf.common.contract.fsm.actions.impl.UploadZipToVcenterAction.execute(UploadZipToVcenterAction.java:72)
        ... 32 common frames omitted

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

SDDC Manager simplifies the deployment and management of VMware Cloud Foundation (VCF). During cluster creation, it heavily depends on vCenter Server to manage the underlying ESXi hosts and ensure they are at the correct version and patch level. The vCenter Server's Update Manager (or vSphere Lifecycle Manager in newer versions) uses patch repositories to access ESXi patches. The default configuration for many vCenter Servers might still point to the old VMware online depot. Since the change to Broadcom's infrastructure, this default configuration is no longer valid.

Resolution

Configuring vCenter Server to Use a Token-Based Custom Patch Depot Instead of the Legacy VMware Depot

Obtain a Download Token:
- Download a Token from the Broadcom Support Portal by following the steps outlined in the "VCF Authenticated Downloads Configuration Update Instructions." This token is required to access the Broadcom patch repositories.
- If you have already downloaded the token from the Broadcom Support Portal, proceed to the next steps.
Disable Default Repositories

Note: Scripted methods to update the URLs are available in Authenticated Download Configuration Update Script.
- Log into vSphere Client
- On the vSphere Client Home page, navigate to "Lifecycle Manager."
- Click on the "Settings" tab.
- Under the "Settings" tab, go to "Administration" → "Patch Setup."
- Disable the default vLCM URLs by clicking the "Disable" button.
  
  Note: If the vCenter Server is running on 7.0 U3c (build 19234570) or 8.0 GA versions (8.0GA, 8.0a, 8.0b & 8.0c), configuring the Custom URLs mentioned below will fail.
  Please apply KB Resetting the VMware Update Manager Database before proceeding with the following steps if the vCenter Server is on these impacted versions.
Add Broadcom Repository URLs:

Add the following URLs one by one by clicking "NEW". Replace <Download Token> with the actual downloaded token ID:
- - https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml
  - https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml
  - https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/iovp-main/vmw-depot-index.xml
  - https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml
Restart Update Manager Service:
- Establish an SSH session to the vCenter Server appliance using root account credentials.
- Execute the following command to restart the Update Manager service:
```
service-control --restart vmware-updatemgr
```
Download Content/Sync Updates:
- Return to the vSphere Client.
- Navigate to the Home page → Lifecycle Manager.
- Select "ACTIONS" → "Updates" → "Sync updates." This will force vCenter Server to download the new patch metadata from the Broadcom repositories.

Verification Steps

After performing the "Sync Updates" action, verify that the patch metadata has been successfully downloaded by examining the Last Check Time and the number of updates available in Lifecycle Manager. If the sync fails, double-check the Download Token and Broadcom repository URLs.

Retry Cluster Creation in SDDC Manager:

To resume the cluster creation process after resolving the initial failure:

Log in to the SDDC Manager Web Console.
Locate the failed Cluster Creation workflow or task.
Restart the workflow or task.