The vMotion/Creation of VM returning the error: "The object 'vim.Datastore:datastore-xxx' has already been deleted or has not been completely created"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• VM Power-On/vMotion fails with the error via Web Client
• Powering on the VM directly from the ESXi host is successful

vpxd log:

2020-03-13T00:08:36.282+02:00 info vpxd[05345] [Originator@6876 sub=vpxLro opID=k7p7ca66-49177-auto-11y2-h5:70003351-3d] [VpxLRO] -- BEGIN session[523e9b57-e829-b17f-88a2-57a4d644f349]52dfa52b-ac80-af14-9fer -- vim.vm.check.ProvisioningChecker.checkRelocate -- 523e9b57-e829-b17f-88a2-57a4d644f349(5213fa69-cf83-4c9d-605b-40558b5e2757)
2020-03-13T00:08:36.283+02:00 info vpxd[05345] [Originator@6876 sub=vpxLro opID=k7p7ca66-49177-auto-11y2-h5:70003351-3d] [VpxLRO] -- FINISH session[523e9b57-e829-b17f-88a2-57a4d644f349]52dfa52b-ac80-af14-9
2020-03-13T00:08:36.283+02:00 info vpxd[05345] [Originator@6876 sub=Default opID=k7p7ca66-49177-auto-11y2-h5:70003351-3d] [VpxLRO] -- ERROR session[523e9b57-e829-b17f-88a2-57a4d644f349]52dfa52b-ac80-af14-9ker -- vim.vm.check.ProvisioningChecker.checkRelocate: vmodl.fault.ManagedObjectNotFound:
--> Result:
--> (vmodl.fault.ManagedObjectNotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    obj = 'vim.Datastore:datastore-216'
-->    msg = ""
--> }
--> Args:
-->
--> Arg vm:
--> 'vim.VirtualMachine:a633b84c-8189-4f7d-95ba-206ef08bcc10:vm-68698'
--> Arg spec:
--> (vim.vm.RelocateSpec) {
-->    service = (vim.ServiceLocator) null,
-->    folder = <unset>,
-->    datastore = <unset>,
-->    diskMoveType = <unset>,
-->    pool = <unset>,
-->    host = 'vim.HostSystem:a633b84c-8189-4f7d-95ba-206ef08bcc10:host-259',
-->    disk = <unset>,
-->    transform = <unset>,
-->    deviceChange = <unset>,
-->    profile = <unset>,
-->    cryptoSpec = (vim.encryption.CryptoSpec) null

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.7.x

Cause

The issue is caused due to the stale entry under the vpx_host_text table on the vCenter Server Database

Resolution

Important Note: Please ensure that a fresh backup of the vCenter Server Appliance has been created before proceeding with the steps below. If the vCenter uses an external database, please do also make to have a fresh database backup.

Furthermore please ensure that the datastore is not in use for any virtual machine.

1. Log in to the vCenter server appliance via ssh
2. Stop the VMware vCenter Server and VMware Content Library Service

   # service-control --stop vmware-vpxd
   # service-control --stop vmware-content-library

3. Log in to the vCenter Server database using the following command

   # /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

4. Find the datastore ID by running this query in the psql shell (replace <datastore_name> with the name of the datastore) :

   select id from VPX_ENTITY where name = '<datastore_name>';

5. Run these queries to verify if the datastore has been removed from the database tables (replace <ID> with the ID found in step 5):

   select * from VPX_DS_ASSIGNMENT where DS_ID=<ID>;
   select * from VPX_VM_DS_SPACE where DS_ID=<ID>;
   select * from VPX_DATASTORE where ID=<ID>;
   select * from VPX_ENTITY where ID=<ID>;
   select * from VPX_HOST_TEXT where value like '%<ID>%';

6. Now run the following 5 queries to delete the datastore entries (again. replace <ID> with the ID of the datastore):

   delete from VPX_DS_ASSIGNMENT where DS_ID=<ID>;
   delete from VPX_VM_DS_SPACE where DS_ID=<ID>;
   delete from VPX_DATASTORE where ID=<ID>;
   delete from VPX_ENTITY where ID=<ID>;
   delete from VPX_HOST_TEXT where value like '%<ID>%';

    
7. Repeat the queries from step 5 to verify that all entries related to the datastore have been removed from the database. If the removal was successful, these queries do not display any results. 
8. Exit psql with "\q" and start the VMware vCenter Server and VMware Content Library Service: 

   # service-control --start vmware-vpxd
   # service-control --start vmware-content-library