After NSX upgrade, vMotion migrations fail and manual vmotion network compatibility shows compatibility issue, "network is under maintenance"

Products

• VMware NSX

Issue/Introduction

• This issue can come up during an NSX in-place upgrade, resulting in host upgrade failure that must be resolved to continue the upgrade process. It can also happen without the hosts failing the NSX upgrade steps but still manifesting in vmotion migrations failing among host in an NSX managed cluster.
• Manual VM vMotion setup during step 3, "Select networks", shows a node compatibility issue when the source host is affected, that states: "Currently connected network interface" 'Network adapter #' cannot use network '<host switch>', because "the network is under maintenance".
  
  
  
• For migrations to an affected host, Compatibility Issues state that a network interface is not accessible and "the network is under maintenance" as well. 
  

Environment

VMware NSX

Cause

The NSX upgrade process puts the host's DVS into an upgrade state, as required. The problem comes up due to failure to complete  final upgrade steps, typically due to external factors such as temporary connection issues between NSX Manager and the hosts or resource contention. 

Resolution

Follow steps from "In-place Upgrade Fails" documentation: In-place Upgrade Fails
 

1. Log in to vCenter Server and place the host in question in maintenance mode.
   *NOTE: If you are in the middle of an upgrade, you will NOT be able to place the host in maintenance mode as you cannot VMotion the VMs off of the host in question. You can run the next command without placing in maintenance mode but we recommend it running during a Maintenance Window. 
2. Verify the issue by running the following command on the ESX host shell:
   nsxcli -c get host-switch upgrade-status
   
3. Use the following command to clear the upgrade status flag on the host:
   nsxcli -c set host-switch upgrade-status false
4. Repeat the steps above for any other hosts in the cluster showing similar symptoms.
5. If these steps are being taken to resolve an NSX upgrade failure, then complete the above steps followed by placing the host in Maintenance mode (migrate VMs off the host now that it is possible) and reboot the host.