vMotion on vSAN Cluster fails with error "The object or item referred to could not be found"

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Its a vSAN Cluster with 2-node (and Witness) running on 8.0 U3.
  
  
• vMotion of any VM fails with the same error:
  "The object or item referred to could not be found"
  
  
• vSphere HA on the ESXi hosts report error:
  "HA detected a possible host failure" 
  
  

• vmkernel of both hosts reported unicast host connection issue:

  2025-06-05T03:35:21.840Z In (182) vmkernel: cpu32: 2098622) CMMDSNet: CMMDSNetSendtoUnicastChannels : 1667: Throttled: 521df3b7-05f2-89bc-0432-0523ab142a85: Failed to send to unicast host '#.#.#.#: 12321' on iface '#.#.#.#': Host is down.
2025-06-05T03:35:27.840Z In (182) vmkernel: cpu32:2098622) CMMDSNet: CMMDSNetSendtoUnicastChannels: 1667: Throttled: 521df3b7-05f2-89bc-0432-0523ab142a85: Failed to send to unicast host '#.#.#.#:12321' on iface "#.#.#.#' : Host is down.
2025-06-05T03:35:32.840Z In (182) vmkernel: cpu32:2098622) CMMDSNet: CMMDSNetSendtoUnicastChannels: 1667: Throttled: 521df3b7-05f2-89bc-0432-0523ab142a85: Failed to send to unicast host '#.#.#.#:12321' on iface '#.#.#.#': Host is down.
  
  

• vSAN hosts reported cluster partitioning under Skyline Health in vSphere Client.

Environment

VMware vSAN 8.x

Cause

• Failed vmnic  on one of the vSAN data node caused Cluster partitioning.
  
  [user@ESXi :~ ] esxcfg-nics -1
  vmnic4 0000:98:00.0 bnxtnet Down OMbps Half ##:##:##.##:##:## 1500 Broadcom BCM57416 NetXtreme-E 10GBASE-T RDMA Ethernet Controller
vmnic5 0000:98:00.1 bnxtnet Up 10000Mbps Full ##:##:##.##:##:## 1500 Broadcom BCM57416 NetXtreme-E 10GBASE-T RDMA Ethernet Controller
  
  
• The same vmnic is used for vMotion and vSAN traffic. 
  
  
• In spite of a stand-by vmnic available, it could not restore the vSAN and vMotion connectivity due to following network configuration/limitation in this environment:
  
  vmnicX of one vSAN node would only communicate to vmnicX of another vSAN node. 
  vmnicY of one vSAN node would only communicate to vmnicY of another vSAN node.
  
  
• Therefore, with vmnicX down on one host and vmnicY taking over, it would not communicate to other host which still as vmnicX as the active adapter.

Resolution

 

1. Navigate to affected ESXi host > Configure > Networking > Virtual switches > Select the vSwitch hosting vSAN vmkernel adapter > Click on the (...) against the PortGroup:
   
   
   
   
2. Click on Edit Settings.
   
   
3. 'Teaming and failover' tab in the wizard.
   
   
4. Make faulty vmnicX as Stand-by and functional vmnicY as Active.
   
   
5. Validate vSAN cluster partitioning issue running 'Retest'.
   
   
6. Once done, test out vMotion of the virtual machines on the cluster.