Packet drops on VMs during DVS upgrade from 6.5 to 7.0.3

Products

• VMware vCenter Server

Issue/Introduction

Network drops are seen on virtual machines attached to the DVS when the upgrade is done from 6.5 to later version.

Cause

When the DVS is upgraded from 6.5 to 7.0.3, the host-switch hotswaps from DVS 6.5 module(etherswitch) to DVS 7.0.3 module(cswitch).

The following actions occur:
  
   -> Ports on the host switch are blocked.
   -> The host-switch is moved from DVS 6.5 module(etherswitch) to DVS 7.0.3 module(cswitch).
   -> The ports are connected back and re-enabled.

We should see minimal downtime when hotswapping from etherswitch to cswitch as the ports are temporarily blocked.

Known bug fixes in datapath around the DVS upgrade hotswap are present in ESX 7.0.3 release. This has been tested locally and we have not encountered issues.

Resolution

The following steps can be taken as best practices and recommendations to reduce the impact on production:-

1. Let this DVS upgrade be done during a maintenance window.

2. Disable DRS on all the clusters that will undergo this upgrade to ensure there are no vMotion at the same time as the upgrade. It can be enabled once the upgrade finishes. This step is important.

3. If an ESX host has two DVSs with one containing a management network and the other one containing regular workloads, please migrate the DVS containing regular workloads first so that even if something goes wrong with the upgrade (ideally shouldn't), we will have access to the ESX host through management DVS.

The worst-case scenario with a DVS upgrade is, that we lose network connectivity to the DVS on a host. If management vmknic is on the same DVS, this may force us to access the host through the console and/or perform a reboot. But we don't have any known issues that could lead to this.

4. Upgrade one DVS at a time.

5. If there is an issue, please collect VC and ESX support bundles immediately rather than collecting them later.

6. Additionally, please follow the below steps :

(2) Schedule a maintenance window for the DVS upgrade.
(3) Do not perform VM operation during the DVS upgrade, which could potentially lead to dvport allocation, and movement between hosts.
(4) Do not perform DVS/DVPG operation during the DVS upgrade.

Note:-Post upgrade if the issues like "The vSphere Distributed Switch configuration on some hosts differed from that of the vCenter Server." are seen, open the case with VMware(Broadcom).For more information, refer the following KB article:-

https://knowledge.broadcom.com/external/article/311368/esxi-hosts-out-of-sync-on-a-vds-after-mi.html