Communication failure on ESXi hosts after manual VDS recreation
search cancel

Communication failure on ESXi hosts after manual VDS recreation

book

Article ID: 438186

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

The VDS may have been manually recreated:

  • Due to real or perceived underlying problems with the VDS. 
  • After a failed upgrade, where restoring from an exported configuration file is not an option. 
  • When needing to revert to an earlier VDS version, since any downgrade of the VDS is not possible.

Symptoms may include:

  • VMs being unable to ping their default gateway, a VM on another Host, or a VM on the same Host using the same portgroup
  • When logged in as root user on an affected host, running esxcli command "net-dvs -l" returns the following line, which can be found under the "host properties" section of the VDS:
    com.vmware.common.host.volatile.dvsID = <VDS_ID> ,       propType = RUNTIME
  • Log output similar to the below can be found in the /var/log/vmware/vpxd/vpxd.log file within the vCenter Server:

<TIME> error vpxd[05721] [Originator@6876 sub=coredvsSync opID=########-1284361-auto-####-h5:########-9f-########] Dvs [<DVS_ID/Name>] port [###] does not exist; ignored update.
<TIME> warning vpxd[05749] [Originator@6876 sub=coredvsSync opID=HB-host-####@2####-5b######] Portgroup [<dvportgroup_ID>] should not be on the host [<ESXi_host_FQDN>]

Collecting diagnostic information for VMware ESXi using vSphere Client

 

 

 

Environment

VMware vCenter Server 8.x
VMware vSphere ESXi 8.x

Cause

This is typically caused by a synchronization failure between the host and vCenter Server. This generally occurs in the following scenarios:

  • Improper Host Removal: If an ESXi host is removed from the vCenter inventory while still connected to a VDS, or if the removal process is interrupted, the host may retain its local VDS configuration. A clean removal requires vCenter to instruct the host to delete its local proxy switch.
  • vCenter Unavailability: Making networking changes, such as migrating management VMkernel ports to a Standard Switch while the vCenter Server is offline or unreachable prevents the host from receiving the command to decommission the DVS components.
  • Residual NSX Configuration: After uninstalling NSX or migrating from N-VDS to VDS, stale configuration entries or properties (e.g., com.vmware.netoverlay) can persist in the DVS metadata, often requiring a manual cleanup script or a host reboot.
  • Config Store Mismatches: In vSphere 8.x, references to physically removed NICs or uplinks can remain in the host Config Store, leading to mismatches with the active DVS state and potentially causing hostd service crashes.

Resolution

The below steps describe the process of removing and replacing the current Virtual Distributed Switch: 

  1. Move vmkernel ports and vmnic's from the DVS to a standard switch on the individual hosts.
    1. Follow KB: Migrating ESXi host networking from Distributed Switch (vDS) to Standard Switch (vSS) via CLI to move all VMkernel ports and vmnic's from the DVS to a standard switch, one host at a time.  
    2. Remove ESXi hosts from DVS:
      Navigate to the Networking inventory in the vSphere Client -------> Select your distributed switch -------> Right-click your distributed switch and select "Add and Manage Hosts" -------> Select "Remove host" and click "Next" ------> Select the ESXi host to remove ------> Click "Next" twice.

  2. Check for stale/orphaned VDS objects or dependencies and delete them along with the remaining DVS data:
    1. Run as root user on each ESXi host:
      #esxcfg-vswitch -l
      • If you still see the DVS listed, run this command to delete it:
        #net-dvs -d <DVS_name>
    2. If you don't see the name of your DVS, but instead see a DvsPortset listed in it's place, (see below for example output) run this command to delete it:
      #vsish -e set /net/portsets/DvsPortset-0/destroy destroy  (The DvsPortset # could be different, but it's usually DvsPortset-0))

      Listing failed for DVSwitch: DvsPortset-0, Error: Unable to get MTU: Status(bad0007) = Bad parameter
    3. Delete the DVS database file from each ESXi host with command:
      #rm -f /etc/vmware/dvsdata.db
    4. Reboot each host

  3. Remove the DVS:
    1. Delete every port group that had been created for that DVS
    2. Delete the old DVS.

  4. Create a new DVS and add Hosts.  Follow Migrating from Standard to Distributed vSwitch to:
    1. Create a new distributed switch.
    2. Create portgroups.
    3. Migrate Management and other networks from the standard switch.
    4. Migrate networking to the new DVS by adding Hosts

  5. Test network connectivity through new DVS:
       A. Check the output of running "net-dvs -l" from the ESXcli.  "com.vmware.common.host.volatile.dvsID" should no longer be present under host properties.
       B. Test connectivity between VM's connected to the new VDS. 

Subscribe to this article to be updated on fix status: KB 275360