ESXi host fails with PSOD (purple screen of death) - Failed to unload module nsxt-vdrb after NSX-T upgrade

Products

• VMware Cloud Foundation
• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

- ESXi Host fails with PSOD (purple screen of death) - Failed to unload module nsxt-vdrb after NSX-T upgrade
- The PSOD backtrace will have similar entries as shown below

VMware ESXi 7.0.0 [Releasebuild-16324942 x86_64]
PF Exception 14 in world 2099166:VdrFLCleaner IP 0x420007c838c4 addr 0x451b038a0feb
PCPU37:2099166/VdrFLCleaner
Code start:0x420006800000 VMK uptime:136:11:11:00.157
0x451af299becd0:[0x420007c838c4]VdrFLEntryDeleteLocked@(nsxt-vdrb-16404614)#,None.+0x8 stack:0x0
0x451af299becd0:[0x420007c838c4]VdrFLCleanerByTable@(nsxt-vdrb-16404614)#<None>+0xc2 stack:0x41ffd5c009e0
0x451af299becd0:[0x420007c838c4]VdrFLCleanerProc@(nsxt-vdrb-16404614)#<None>+0xc2 stack:0x41ffd5c009e0
0x451af299becd0:[0x420007c838c4]vmkWorldFunc@vmkernel#nover+0x4a stack:0x420006919ea1
0x451af299becd0:[0x420007c838c4]CpuSched_startWorld@vmkernel#nover+0x4a stack:0x420006919ea1
0x451af299becd0:[0x420007c838c4]Debug_isInitialized@vmkernel#nover+0x4a stack:0x420006919ea1
2020-11-27T14:33:41.288Z cpu67:21802840)Mod:5251:Failed to unload module nsxt-vdrb-16404614,since its consumed resource count is 1.Waiting...
 
A sample screenshot is shown below:

- In vmkernel.log you will see similar to below

2020-09-25T05:40:14.997Z cpu49:2230556)Mod: 5059: Unloading module <nsxt-vdrb-16404614> ...
2020-09-25T05:40:14.997Z cpu49:2230556)nsx_vdrb: cleanup_module:2182: [nsx@6876 comp="nsx-esx" subcomp="vdrb-16556498"]SYS:Starting cleanup
2020-09-25T06:45:52.510Z cpu90:2105602)Mod: 5059: Unloading module <nsxt-vdrb-16404614> ...
2020-09-25T06:45:52.510Z cpu90:2105602)nsx_vdrb: cleanup_module:2182: [nsx@6876 comp="nsx-esx" subcomp="vdrb-16404614"]SYS:Starting cleanup
2020-09-25T06:45:52.510Z cpu90:2105602)nsx_vdrb: VdrFLCleanup:1151: [nsx@6876 comp="nsx-esx" subcomp="vdrb-16404614"]SYS:Into vdrFLCleanup()

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment

Environment

VMware vSphere ESXi 7.0.0
VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware vSphere ESXi 6.7

Cause

This  issue  occurs on servers with large number of pcpus causing NSX uninstall and DEL-NSX both failing.

Resolution

This issue is resolved in NSX-T version 3.1.1.

Workaround:
Disable nsxt-vdrb-16404614 by running  below command in ESXi host
esxcfg-module -d nsxt-vdrb-16404614 and reboot before upgrade.

NOTE: Here the version build 16404614 can change depending on your environment.

Additional Information

Impact/Risks:
Requires manual intervention.