vSphere HA Configuration Error: 'The object 'vim.Datastore:datastore-#####' has already been deleted or has not been completely created'

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• When attempting to configure vSphere High Availability (HA) for a cluster, you might encounter the following error:

     “The object 'vim.Datastore:datastore-#####' has already been deleted or has not been completely created.”

• As a result, the HA configuration fails to complete, and hosts within the cluster show a "Configuration Issue" warning.
  
  
• In the vCenter server logs, you may observe an error similar to the following from /var/log/vmware/vpxd/vpxd.log:
  
  
  

  YYYY-MM-DDThh:mm:ss info vpxd[7FEAA8B29700] [Originator@6876 sub=DAS opID=DrsHaConfigWithTOCDialog-apply-17575124-ngc:70730085-da-09] [VpxdDasConfigLRO::Cleanup] Number of unprotected vms: 5
  YYYY-MM-DDThh:mm:ss info vpxd[7FEAA8B29700] [Originator@6876 sub=vpxLro opID=DrsHaConfigWithTOCDialog-apply-17575124-ngc:70730085-da-09] [VpxLRO] -- FINISH task-236530
  YYYY-MM-DDThh:mm:ss info vpxd[7FEAA8B29700] [Originator@6876 sub=Default opID=DrsHaConfigWithTOCDialog-apply-17575124-ngc:70730085-da-09] [VpxLRO] -- ERROR task-236530 -- <FQDN OF ESXI HOST> -- DasConfig.ConfigureHost: vmodl.fault.ManagedObjectNotFound:
  --> Result:
  --> (vmodl.fault.ManagedObjectNotFound) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    obj = 'vim.Datastore:######-2##9-4##b-a##c-20#########:datastore-###'
  -->    msg = ""
  --> }
  --> Arg
  
  YYYY-MM-DDThh:mm:ss  error vpxd[7FEAAB57D700] [Originator@6876 sub=DAS opID=DrsHaConfigWithTOCDialog-apply-17575124-ngc:70730085-da-04] EnableDAS failed on host [vim.HostSystem:host-####,<FQDN OF ESXI HOST>]: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(vmodl.fault.ManagedObjectNotFound)
  --> [context]zKq7AVECAAAAAAHCfgAWdnB4ZAAAJGcrbGlidm#####################-apply-17575124-ngc:70730085-da-04] [HostMo::UpdateDasState] VC state for host host-#### (uninitialized -> init error), FDM state (Live -> Live), src of state (null -> null)
  YYYY-MM-DDThh:mm:ss  info vpxd[7FEAA8E2F700] [Originator@6876 sub=DAS opID=DrsHaConfigWithTOCDialog-apply-17575124-ngc:70730085-da-08] [VpxdDasConfigLRO::Cleanup] Number of unprotected vms: 7
  YYYY-MM-DDThh:mm:ss info vpxd[7FEAA8E2F700] [Originator@6876 sub=vpxLro opID=DrsHaConfigWithTOCDialog-apply-17575124-ngc:70730085-da-08] [VpxLRO] -- FINISH task-236529
  YYYY-MM-DDThh:mm:ss info vpxd[7FEAA8E2F700] [Originator@6876 sub=Default opID=DrsHaConfigWithTOCDialog-apply-17575124-ngc:70730085-da-08] [VpxLRO] -- ERROR task-236529 -- <FQDN OF ESXI HOST> -- DasConfig.ConfigureHost: vmodl.fault.ManagedObjectNotFound:
  --> Result:
  --> (vmodl.fault.ManagedObjectNotFound) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    obj = 'vim.Datastore:######-2##9-4##b-a##c-20#########:datastore-###'
  -->    msg = ""
  --> }
  --> Args:

Cause

The root cause of the issue is that the referenced datastore, which was previously used in the HA configuration, has been deleted but still remains in the HA configuration.

The HA agent on the ESXi host contains stale references to a deleted datastore that was previously used for storing Heartbeat Datastores. If the datastore is removed without making the necessary changes to the HA configuration, the error occurs when attempting to reconfigure HA.

Resolution

To resolve this issue and successfully configure HA for the cluster, follow these steps:

1. Disable HA for the cluster:

   • Navigate to the cluster settings in vCenter.

   • Disable HA by unchecking the HA configuration options.

2. Re-scan VMFS/datastores for the entire cluster:

   • Perform a re-scan of the VMFS/datastores for the entire cluster to ensure that all datastores are detected correctly.

3. Reboot the vCenter server:

   • Restart the vCenter server to clear any lingering references to the deleted datastore and refresh the system.

4. Wait for 10-15 minutes:

   • Allow enough time for vCenter services to fully come up and stabilize after the reboot.

5. Reconfigure HA:

   • Re-attempt to configure vSphere HA for the cluster.

6. If HA reconfiguration fails, follow the alternative method:

   • Go to the vCenter console and click on your Cluster.

   • Click on the ‘Configure’ tab.

   • Navigate to the ‘vSphere Availability’ section.

   • Click ‘Edit’ to modify HA settings.

   • Under ‘Heartbeat Datastores’, select Use datastores only from the specified list.

   • Select any datastore that all your hosts have access to and click OK to apply changes.