VCF Workload Domain Creation Fails at Remediate ESXi Hosts to be compliant with Cluster’s Image

Products

VMware SDDC Manager VMware Cloud Foundation VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

When attempting to create a new Workload Domain (WLD) or add a cluster in VMware Cloud Foundation (VCF), the task fails at the stage "Remediate ESXi Hosts to be compliant with Cluster’s Image".

The log /var/log/vmware/vcf/domainmanager/domainmanager.log in SDDC Manager indicates that applying the cluster personality failed. You may see a warning regarding InteropBundleUpdateRequired, but the critical status is VLCM_REMEDIATE_PERSONALITY_FAILED

ERROR [vcf_dm,6964............,9333] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-20]  [######] VLCM_REMEDIATE_PERSONALITY_FAILED Applying personality to cluster domain-c# failed with error: ApplyStatus (com.vmware.esx.settings.clusters.software.apply_status) => {
    status = SKIPPED,
    progress = <null>,
    startTime = yyyy-mm-ddThh:mm:ss.380Z[GMT],
    endTime = yyyy-mm-ddThh:mm:ss.131Z[GMT],
    notifications = Notifications (com.vmware.esx.settings.notifications) => {
        info = <null>,
        warnings = [Notification (com.vmware.esx.settings.notification) => {
    type = WARNING,
    id = com.vmware.vcIntegrity.lifecycle.scl.InteropBundleUpdateRequired,
    time = yyyy-mm-ddThh:mm:ss.127Z[GMT],
    message = LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
        id = com.vmware.vcIntegrity.lifecycle.scl.InteropBundleUpdateRequired,
        defaultMessage = Software compatibility cannot be determined for release 9.0.1.0.24957456, 9.0.1.0.24957454. It can be either a security patch or an out-of-date compatibility data bunlde having timestamp yyyy-mm-ddThh:mm:ss.181000. Firstly, please ensure the software compatibility data bundle is up-to-date. Please note that no software compatibility checks are performed on security patch
es.,
        args = [9.0.1.0.24957456, 9.0.1.0.24957454, yyyy-mm-ddThh:mm:ss.181000],
        params = <null>,
        localized = Software compatibility cannot be determined for release 9.0.1.0.24957456, 9.0.1.0.24957454. It can be either a security patch or an out-of-date compatibility data bunlde having timestamp yyyy-mm-ddThh:mm:ss.181000. Firstly, please ensure the software compatibility data bundle is up-to-date. Please note that no software compatibility checks are performed on security patches.
    },
    resolution = LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
        id = com.vmware.vcIntegrity.lifecycle.health.sclcheck.outdated_bundle.resolution,
        defaultMessage = Sync or upload up-to-date compatibility data bundle from Lifecycle Manager.,
        args = [],
        params = <null>,
        localized = Sync or upload up-to-date compatibility data bundle from Lifecycle Manager.
    },
    originator = <null>,
    retriable = <null>
}],
        errors = <null>
    }
}

Error in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log in vCenter server identifies a critical communication failure: EsxImage.DepotConnectError ... urlopen error timed out

error vmware-vum-server[#####] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask opID=789b............] [ClusterScanTask, 905] Task:com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask ID:522b................ Failed to get host : esxi01.example.com (host-##) image result. Errors : Error:
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.EsxImage.DepotConnectError<Failed to connect to depot: 'URL: 'http://wldvcenter.example.com:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-167__index__.xml' Error: '<urlopen error timed out>''>
--> ;default compliance will be substituted.

Environment

VMware Cloud Foundation

VMware vCenter Server

VMware vSphere ESXi

Cause

The issue is caused by a firewall or network restriction blocking TCP Port 9084.
During the remediation process, the ESXi hosts must connect to the vCenter Server (specifically the vSphere Update Manager service) to download metadata and upgrades. This communication occurs over port 9084. If the ESXi host cannot reach http://<vCenter_FQDN>:9084, the remediation task times out and fails

Resolution

To resolve this issue, ensure that traffic is allowed on port 9084 between the ESXi hosts in the cluster and the vCenter Server managing that Workload Domain.

Refer VMware Ports and Protocols

Validate Connectivity
1. SSH into one of the affected ESXi hosts with root user.
2. Attempt to reach the vCenter on port 9084 using nc command
```
nc -z <vCenter_FQDN_or_IP> 9084
```
  Success: Connection to vcsa01.example.com 9084 port [tcp/*] succeeded!
  Failure: Connection timed out / Refused
Update Network Rules
1. Engage your Network Administrator to update the physical firewall or ACLs.
2. Allow TCP 9084 from ESXi Management Network to vCenter Server.
Retry the Operation
1. Once connectivity is verified, return to the SDDC Manager UI.
2. Navigate to the Tasks panel.
3. Restart the failed Workload Domain creation task.