ESXi hosts are overcommited with running VMs during NSX upgrade

Products

• VMware NSX

Issue/Introduction

Symptoms:

During an NSX upgrade, many ESXi hosts in a cluster are evacuated and placed in maintenance mode. The remaining virtual machines are then placed onto the minimum number of ESXi hosts. When this occurs, you see these symptoms:

• You may see these working ESXi hosts spike in CPU usage.
• Slow performance or disconnections are experienced to the running virtual machines.
• You are allowing fully-automated DRS to coordinate the upgrade.

Environment

VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.3.x

Resolution

This is a known issue when using fully-automated DRS at a more aggressive level during NSX upgrade.

There are two ways you can work around this to prevent it from occurring in the future:

1. Set DRS to Manual on the cluster before upgrading.
2. Edit the eam-vim.properties configuration file on the vCenter Server.

Notes:

• EAM, by default, requests that DRS place all hosts into Maintenance Mode. Thus, the drs.demandCapacityRatio parameter is set to "100" in the eam-vim.properties file.
  • For Windows, the location of the eam-vim.properties file: C:\ProgramData\VMware\vCenterServer\cfg\vmware-eam.
  • For Linux or VCSA: /etc/vmware-eam (use "vi" to edit the file).
• You can change the "100" to a lower number so that EAM requests fewer hosts be placed into maintenance mode. So, if you set the drs.demandCapacityRatio=75, then EAM will requests only 75% of hosts get placed into maintenance mode at one time. This prevents virtual machines from being placed on the minimum number of hosts and prevent any performance impact during the upgrade. Once this is completed, restart the EAM Service.
• This change is persistent after vCenter reboots and minor upgrades (vCenter 6.0 Update 1 to 6.0 Update 3). The change is not persistent after major upgrades of vCenter Server such as version 6.0 to 6.5.