vSAN cluster restart fails at "Re-enable HA on this cluster" with task contention or DNS errors
search cancel

vSAN cluster restart fails at "Re-enable HA on this cluster" with task contention or DNS errors

book

Article ID: 425933

calendar_today

Updated On:

Products

VMware vSAN VMware vCenter Server 8.0 VMware vSphere ESXi

Issue/Introduction

  • When restarting a vSAN cluster using the Cluster Shutdown and Startup Wizard, the operation fails at the following step:
    • Re-enable HA on this cluster: Wait for HA enable failed
  • Ensure external firewalls between vCenter and ESXi hosts allow inbound connection to vCenter from the ESXi hosts over port 9084. If the firewall does not have port 9084 open, tasks such as compliance scans, upgrades, or enabling vSphere HA may fail. 
  • The cluster restart does not complete. Even after clicking **Resume Restart**, the task may fail again. 

  • Additional logs in `/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log` may show:
    • A failure occurred when starting a host compliance check operation on host  : Error: com.vmware.vapi.std.errors.not_allowed_in_current_state
      Messages: com.vmware.esx.task.contention <Another task is in progress. Please retry later.>

  • On ESXi hosts, `/var/run/log/lifecycle.log` may report:
    • DepotCollection:### Downloading depot index.xml from http://<VC_FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-13__index__.xml
      Downloader:### Opening http://<VC_FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-13__index__.xml for download
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 9 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 8 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 7 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 6 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 5 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 4 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 3 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 2 retry left...
      Downloader:### Download failed: <urlopen error [Errno -3] Temporary failure in name resolution>, 1 retry left...
      DepotCollection:### Could not download from depot at http://<VC_FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-13__index__.xml, skipping (('http://<VC_FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-13__index__.xml', '', '<urlopen error [Errno -3] Temporary failure in name resolution>'))

Environment

  • VMware vCenter Server 8.X
  • VMware vSphere ESXi 8.X
  • VMware vSAN 8.X 

Cause

  • During the Restart operation, the wizard attempts to re-enable vSphere HA. ESXi hosts must resolve the vCenter FQDN to connect to the vCenter Lifecycle Manager (vLCM) via TCP port 9084 to download metadata. 
  • If DNS is unavailable (e.g., the DNS server is a VM on the same vSAN cluster that is not yet powered on), the hosts cannot resolve the vCenter FQDN. This causes the HA metadata download to fail.
  • In some cases, this failure leaves behind stale background tasks, resulting in `task.contention` errors during subsequent retries.

Resolution

  1. Ensure that DNS services are available and functioning before restarting the vSAN cluster.

  2. Verify that ESXi hosts can resolve the vCenter FQDN.

    • nslookup <vCenter-FQDN>
  3. If DNS cannot be made available immediately, use one of the following workarounds on every ESXi host in the cluster:

    • Non-persistent workaround:

      • Add an entry directly to /etc/hosts. This change does not persist across ESXi host reboots.

        <vCenter_IP_Address> <vCenter_FQDN>

         

    • Persistent workaround:

      • Use the following command to add a host entry that persists across ESXi host reboots:

        • esxcli network ip hosts add --hostname <VC_FQDN> --ip <VC_IP>
  4. If com.vmware.esx.task.contention errors persist, reboot the affected ESXi hosts to clear stale background tasks and release the lock.

  5. Click Resume Restart or retry the operation in the vSphere Client.

  6. If the above steps should continue to fail, perform the tasks listed below
  • Place all three ESXi hosts into Maintenance Mode using the 'No Action' (No data migration) evacuation mode since virtual machines are already powered off.
  • Perform a coordinated reboot of all three hosts in the cluster.
  • Once all hosts are back online, re-initiate the vSAN Cluster Restart/Resume workflow to allow FDM and HA services to initialize correctly.

Additional Information

  • vSphere HA enablement depends on proper DNS name resolution of the vCenter when the vCenter was deployed with FQDN.
  • Using /etc/hosts as a temporary workaround when DNS is unavailable.
  • DNS unavailability at cluster startup time can cause failures even if DNS servers are hosted outside the cluster.

Refer KB article: After enabling Lifecycle Manager and initiating a Compliance Check, vSphere HA fails 

To speak with a customer representative or a Support Engineer see Contact Broadcom Support. Scroll to the bottom of the page and click on your respective region.