vSphere High Availability failover fails on virtual machines due to missing .dvsData file

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

vSphere High Availability (HA) failover fails for random virtual machines and causes the affected virtual machine to stay in powered-off state.

The vSphere HA task fails with the error: vSphere HA virtual machine failover failed.
In the fdm log (log path: /var/log/fdm.log) the following error is observed: vim.fault.PlatformConfigFault and Status (bad0003)= Not found.

YYYY-MM-DDTHH:MM Db(167) Fdm[####380]: [Originator@6876 sub=Hal opID=<Host_moid>:8:########-0] Loading port [19] on dvs [## ## ## a1 b4 ## ## ##-## ## ## ## ## ## ## ##] at location [/vmfs/volumes/vsan:52####################b37/########-####-####-####-########0003/.dvsData/## ## ## a1 b4 ## ## ##-## ## ## ## ## ## ## ##/<dvport_id>]
YYYY-MM-DDTHH:MM In(166) Fdm[####380]: [Originator@6876 sub=vmomi.soapStub[1] opID=<Host_moid>:8:########-0] SOAP request returned HTTP failure; <<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE, h:11, <TCP '127.0.0.1 : 50407'>, <TCP '127.0.0.1 : 8307'>>, /sdk>, method: loadDVPort; code: 500(Internal Server Error); fault: (vim.fault.PlatformConfigFault)
YYYY-MM-DDTHH:MM In(166) Fdm[####325]: --> faultCause = (vmodl.MethodFault) null,
YYYY-MM-DDTHH:MM In(166) Fdm[####325]: --> faultMessage = (vmodl.LocalizableMessage) [
YYYY-MM-DDTHH:MM In(166) Fdm[####325]: --> (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> key = "com.vmware.esx.hostctl.default",
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> arg = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> key = "reason",
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> value = "Failed to open file /vmfs/volumes/vsan:52####################b37/########-####-####-####-########0003/.dvsData/## ## ## a1 b4 ## ## ##-## ## ## ## ## ## ## ##/<dvport_id> Status (bad0003)= Not found"
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> message = "Operation failed, diagnostics report: Failed to open file /vmfs/volumes/vsan:52####################b37/########-####-####-####-########0003/.dvsData/## ## ## a1 b4 ## ## ##-## ## ## ## ## ## ## ##/<dvport_id> Status (bad0003)= Not found"
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> msg = "Received SOAP response fault from [<<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE, h:11, <TCP '127.0.0.1 : 50407'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: loadDVPort
YYYY-MM-DDTHH:MM In(166) In(166) Fdm[####325]: --> An error occurred during host configuration: ."
YYYY-MM-DDTHH:MM In(166) Er(163) Fdm[####380]: [Originator@6876 sub=Default opID=<Host_moid>:8:########-0] Got error invoking loadDVPort on vim.dvs.HostDistributedVirtualSwitchManager:ha-hostdvsmanager: N3Vim5Fault19PlatformConfigFault9ExceptionE(Fault cause: vim.fault.PlatformConfigFault

hostd log (log path: /var/log/hostd.log) reports multiple 'Skip saving dvport' events every 5 minutes after the VM is migrated from Standard switch to Distributed switch.

YYYY-MM-DDTHH:MM Wa(164) Hostd[####413]: [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport <distributed_switch_name>-<port_id> to: filePath empty
None of the hosts in the cluster are able to locate or open the .dvsData metadata files on the shared datastore, hence the virtual machine fails to start on any of the hosts.

Environment

vCenter 9.x
ESX 9.x

Cause

This happens due to missing .dvsData files on storage after VM NIC is migrated from Standard switch to Distributed switch.

Resolution

Broadcom Engineering is currently working on a fix to resolve this issue in future 9.x version.

Workaround:

To resolve this issue, manually force the creation of .dvsData persistence files by toggling the affected virtual machine's network adapter port group. This triggers a re-registration of the port configuration and saves the necessary metadata to the shared datastore.

Log in to the vSphere Client.
Right-click the affected virtual machine and select Edit Settings.
Expand the Network adapter (e.g., Network adapter 1).
From the Network Connection dropdown menu, select a different port group.
Ensure Connected is checked and click OK.
After the virtual machine reconfiguration task completes, right-click the virtual machine again and select Edit Settings.
Change the network connection back to the original port group.
Click OK to finalize.

Verification:

To verify if the workaround was successful without triggering a failover, run the following command on the ESXi host where the virtual machine is currently running:

net-dvs -l

Check the output for the port used by the virtual machine. The com.vmware.common.port.volatile.persist property must contain a valid path to the .dvsData folder on the shared datastore. If this property is missing or empty, the VM will fail to failover.