ESXi host in Not Ready state after major ESX version upgrade

Products

• VMware NSX

Issue/Introduction

There is an NSX VIB version corresponding to each ESXi major version.
If a major ESXi version upgrade takes place, the NSX VIB must also be upgraded.
It is not supported to run an NSX VIB version that is a mismatch for the ESXi version.
In that state there can be compatibility issues and NSX modules may fail to load.
After a major version upgrade of ESXi, the ESXi host will exit from maintenance mode.
At this point EAM will evaluate the host and detect the version change.
EAM will update the agent for this host and push the correct NSX VIB to the host.
If that upgrade should fail e.g. ESXi reports it requires the host to be in Maintenance Mode, then EAM will notify NSX of the failure.
NSX will mark the host in Error state.
At that point manual intervention is required by the user to resolve the issue.

Symptoms:

• NSX Data Center for vSphere environment.
• A major ESXi version upgrade has been performed e.g 6.5 to 6.7, 6.7 to 7.0 etc.
• On the UI, Network & Security - > Installation -> Host Preparation some hosts may be in a Not Ready state.
• VMs connected to Logical Switches running on these hosts may experience network issues.
• ESXi /var/log/hostd.log reports an NSX VIB install failure e.g.

2020-10-02T07:54:33.504Z info hostd[2101210] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 54 : Could not install image profile: MaintenanceMode is required to remove: [VMware_bootbank_esx-nsxv_6.5.0-0.0.11070622]; install: [].

• ESX Agent Manager (EAM) log /var/log/vmware/eam/eam.log on vCenter also reports the failure e.g.

2020-09-30T11:01:58.726Z | INFO | vim-inv-update | VcHostSystem.java | 422 | VcHostSystem(ID: host-1513) changed version from 6.5 to 6.7
2020-09-30T11:04:55.064Z | WARN | host-1513-2 | Retry.java | 103 | Failed to VIB install dry-run at attempt 1 out of 6 Reason: com.vmware.eam.vi.vib.exception.MaintenanceModeException: VIB operation requires the host to be put in maintenance mode
2020-09-30T11:04:55.065Z | ERROR | host-1513-2 | AgentImpl.java | 2041 | AgentImpl(ID:'Agent:589f39fe-7a36-459f-b62e-0eacedc29787:null') VIB installation failed: https://192.168.10.10/bin/vdn/vibs-6.4.4/6.7-11070622/vxlan.zip

• If the ESXi host is rebooted in this state, a failure to load NSX modules may be seen in vmkernel.log e.g.

2020-10-02T07:53:49.278Z cpu44:2101963)ALERT: Failed to load kernel modules: nsx-vdrb
WARNING: Elf: 3144: Kernel based module load of nsx-vdrb failed: Unresolved symbol <ElfRelocateFile failed>

Resolution

This is a known issue affecting NSX Data Center for vSphere.

Workaround:

Ensure DRS is set to Fully Automated on the cluster.
From Network & Security -> Installation -> Host Preparation
Select the host in a Not Ready state and click Resolve
The host will be put in Maintenance Mode and the NSX VIB will be upgraded.