vMotion fails for a specific host. Migration fails with error message "The Object or item referred to could not be found"

Products

• VMware vSphere ESXi
• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware vCenter Server

Issue/Introduction

Unable to vMotion virtual machines from specific Host. The migration fails with error message "The Object or item referred to could not be found"

Environment

• VMware Aria Suite 8.x
• VMware Aria Automation 8.x
• ESXi 8.x
  
• ESX 9.x
• vCenter 8.x
• vCenter 9.x

Cause

When an ESX host is added into vDS and performing VM networking migration, all VMs on that particular Host do not register on dVport correctly, the new port ID does not get assigned to VM network and it continues to use the old port ID and network connection remains stable. 

 /var/log/hostd.log

error hostd[2101236] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled

[LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

 error hostd[2100948] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled

 

info hostd[2101544] [Originator@6876 sub=Solo.Vmomi opID=2f09d043-7f00 user=<user>] Result:

--> (vim.fault.PlatformConfigFault) {

--> faultCause = (vmodl.MethodFault) null,

--> faultMessage = (vmodl.LocalizableMessage) [

--> (vmodl.LocalizableMessage) {

--> key = "com.vmware.esx.hostctl.default",

--> arg = (vmodl.KeyAnyValue) [

--> (vmodl.KeyAnyValue) {

--> key = "reason",

--> value = "Unable to delete DVPort "##" that is in use, use list: example-vm.eth0"

--> }

--> ],

--> message = <unset>

--> }

--> ],

--> text = ""

--> msg = ""

--> }

Resolution

Attempt any of the below options

• Power off Virtual Machine and unregister. Register it back

OR

• Clone the Virtual Machine (Cloned VM should acquire the right port ID)

OR 

• Login to vSphere Client using administrator account.  

1. Select the respective dvSwitch, right click select "Add and Manage Hosts" and click "Manage Host Networking"
2. Select the problematic Host 
3. Skip to "Migrate VM networking". Check the box "Migrate virtual machine networking" to see the VMs on the host.
4. Select "Configure per virtual machine". The list of VMs shows the current port group under one of the columns.
5. Click on "Assign Port Group". It will display a list of Port Groups on the dvSwitch. Select a port group to assign the VM to. If possible, reassign the same port group it was already using. If not possible, assign a different port group. (Assigning a different port group could cause the VM to lose connectivity until you reassign it back.)
6. Click "Next" then "Finish"
7. Assigning the same port group will re-register and will update the network port ID of the VM. You should be able to Migrate the VM after this process. 
8. During the network re-registration, the VM will lose the network for few seconds based on network traffic at that point of time. 
9. You may validate by clicking the dVSwitch => Configure => Settings => Topology. The UI should show you "Port Group" and the VM connection should show you "Green " visible connection to uplinks.

Additional Information

Impact/Risks:

No Impact to VMs connection, only migration fails with error message.

You might tend to see similar issue when initiating this task via "Aria Automation Orchestrator or Standalone Orchestrator" Workflow : relocateVM_Task , failing with error "The object or item referred to could not be found. vDS dvs port *** cannot be found".

The fix for the migrations is to remove the portID that they are passing in the API call as that is causing the error in the migrations.