vMotion fails for a specific hosts. Migration fails with error message "The Object or item referred to could not be found"

Products

• VMware vSphere ESXi
• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Unable to vMotion virtual machines from specific Host. The migration fails with error message "The Object or item referred to could not be found"

Environment

VMware Aria Suite 8.x
VMware Aria Automation 8.x
VMware vSphere ESXi 8.0.x
VMware vCenter Server 8.0.x
VMware vSphere ESXi 7.0.x
VMware vCenter Server 7.0.x

Cause

When the customer adds the ESXi host into vDS and performs VM networking migration, all VM on that particular Host does not register on dVport correctly, the new port ID does not get assigned to VM network and it continues to use the old port ID and network connection remains stable. 

 /var/log/hostd.log

error hostd[2101236] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled

[LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

 error hostd[2100948] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled

 

info hostd[2101544] [Originator@6876 sub=Solo.Vmomi opID=2f09d043-7f00 user=vpxuser] Result:

--> (vim.fault.PlatformConfigFault) {

--> faultCause = (vmodl.MethodFault) null,

--> faultMessage = (vmodl.LocalizableMessage) [

--> (vmodl.LocalizableMessage) {

--> key = "com.vmware.esx.hostctl.default",

--> arg = (vmodl.KeyAnyValue) [

--> (vmodl.KeyAnyValue) {

--> key = "reason",

--> value = "Unable to delete DVPort "" that is in use, use list: VM-NAME.eth0"

--> }

--> ],

--> message = <unset>

--> }

--> ],

--> text = ""

--> msg = ""

--> }

Resolution

Attempt any of options below

• Power off Virtual Machine and unregister. Register it back

OR

• Clone the Virtual Machine and the cloned VM should acquire the right port ID

OR 

• Open the vCenter server UI and login with an administrator account.  

1. Select the respective dvSwitch , => right click select "Add and Manage Hosts" => Manage Host Networking 
2. Select the problematic Host 
3. Skip to "Migrate VM networking". Check the box "Migrate virtual machine networking" to see the VMs on the host.
4. Select "Configure per virtual machine". The list of VMs should show you the current port group under one of the columns.
5. Click on "Assign Port Group". It will display a list of Port Groups on the dvSwitch. Select a port group to assign the VM to. If possible, reassign the same port group it was already using. If not possible, assign a different port group. (Assigning a different port group could cause the VM to lose connectivity until you reassign it back.)
6. Click "Next" then "Finish"
7. Assigning the same port group will re-register and will update the network port ID of the VM. You should be able to Migrate the VM after this process. 
8. During the network re-registration, the VM will lose the network for few seconds based on network traffic at that point of time. 
9. You may validate by clicking the dVSwitch => Configure => Settings => Topology. The UI should show you " Port Group" and the VM connection should show you "Green " visible connection to uplinks.

Additional Information

Impact/Risks:

No Impact to VMs connection, only migration fails with error message.

You might tend to see similar issue when initiating this task via " Aria Automation Orchestrator or Standalone Orchestrator " Workflow : relocateVM_Task , failing with error "The object or item referred to could not be found. vDS dvs port *** cannot be found". 

 

The fix for the migrations was to remove the portID that they were passing in the API call as that was causing the error in the migrations.