vMotion fails for a specific hosts. Migration fails with error message "The Object or item referred to could not be found"

Products

• VMware vSphere ESXi

Issue/Introduction

To troubleshoot the migration of VM across the hosts. To resolve the issue of host vMotion and troubleshoot

Symptoms:

• Issue: Unable to vMotion virtual machines from specific Host. The migrations fails with error message "The Object or item referred to could not be found"

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0

Cause

Inside the vCenter, the Host is already part of cluster. When the customer adds the ESXi host into vDS and perform VM networking migration, all VM on that particular Host does not register on dVport correctly, The new port ID does not get assigned to VM network and it continues to use the old port ID  and network connection remains stable. 

On the ESXi hosts you  will see  below error:

/var/log/hostd.log :

2019-05-14T03:21:53.272Z error hostd[2101236] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled

[LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

2019-05-14T03:22:23.275Z error hostd[2100948] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled

 

2019-05-14T03:18:52.472Z info hostd[2101544] [Originator@6876 sub=Solo.Vmomi opID=2f09d043-7f00 user=vpxuser] Result:

--> (vim.fault.PlatformConfigFault) {

--> faultCause = (vmodl.MethodFault) null,

--> faultMessage = (vmodl.LocalizableMessage) [

--> (vmodl.LocalizableMessage) {

--> key = "com.vmware.esx.hostctl.default",

--> arg = (vmodl.KeyAnyValue) [

--> (vmodl.KeyAnyValue) {

--> key = "reason",

--> value = "Unable to delete DVPort "" that is in use, use list: VM-NAME.eth0"

--> }

--> ],

--> message = <unset>

--> }

--> ],

--> text = ""

--> msg = ""

--> }

Resolution

There are two solutions :
 

1) You need to Power Off Virtual Machine and  Unregister it. Then register the virtual machine again and migrate. You may try creating a new test-VM or deploy a test-VM from template or Clone existing VM to test-VM

OR 

2) Open the vCenter server UI and login with an administrator account. 

• Select the respective dVSwitch  , => right click select "Add and Manage Hosts" => Manage Host Networking 
• Select the problematic Host ( Check Box ) 
• Skip to "Migrate VM networking". Check the box "Migrate virtual machine networking" to see the VMs on the host.
• Select "Configure per virtual machine". The list of VMs should show you the current port group under one of the columns.
• Click on "Assign Port Group". It will display a list of Port Groups on the dVSwitch. Select a port group to assign the VM to. If possible, reassign the same port group it was already using. If not possible, assign a different port group. (Assigning a different port group could cause the VM to lose connectivity until you reassign it back.)
• Click "Next" then "Finish"
• Assigning the same port group will  re-registration of network port ID of the VM and you should be able to Migrate the VM after this process. 
• During the network re-registration,  the VM will loose the network for few seconds based on network traffic at that point of time. 
• You may validate by clicking the dVSwitch => Configure => Settings => Topology. The UI should show you " Port Group"  and the VM connection should show you "Green " visible connection to uplinks 
• We recommend you to try this on non-critical VM and after success you may try on critical-production VM.

Additional Information

Impact/Risks:
No Impact to VMs connection, only migration fails with error message.