Removal of NSX-T Transport node from a cluster fails if SNMP is running on ESXi host

Products

• VMware NSX

Issue/Introduction

Symptoms:

• In the /var/run/log/esxupdate.log, you see entries similar to:

2021-03-11T19:17:19Z esxupdate: 3311819: LiveImageInstaller: DEBUG: output: rm: can't remove '/tardisks/nsx_esx_.v00': Device or resource busy                                                                                                                                
2021-03-11T19:17:19Z esxupdate: 3311819: LiveImageInstaller: INFO: Received error: Error in running rm /tardisks/nsx_esx_.v00: Return code: 1 Output: rm: can't remove '/tardisks/nsx_esx_.v00': Device or resource busy  It is not safe to continue. Please reboot the host i
2021-03-11T19:17:20Z esxupdate: 3311819: LiveImageInstaller: DEBUG: Unmounting nsx_esx_.v00...                                                                                                                             
2021-03-11T19:17:20Z esxupdate: 3311819: vmware.runcommand: INFO: runcommand called with: args = 'rm /tardisks/nsx_esx_.v00', outfile = 'None', returnoutput = 'True', timeout = '0.0'.                                    
2021-03-11T19:17:20Z esxupdate: 3311819: LiveImageInstaller: DEBUG: output: rm: can't remove '/tardisks/nsx_esx_.v00': Device or resource busy
2021-03-11T19:17:20Z esxupdate: 3311819: LiveImageInstaller: INFO: Received error: Error in runnin

 

• In the vmkernel.log, you see entries similar to:

2021-03-11T19:17:12.234Z cpu1:3313169)VisorFSRam: 1164: inode 3296343256006294816               
2021-03-11T19:17:12.234Z cpu1:3313169)VisorFSRam: 1181: ramdisk nsx-idps                        
2021-03-11T19:17:12.234Z cpu1:3313169)VisorFSRam: 1347: for ramdisk nsx-idps                    
2021-03-11T19:17:15.405Z cpu3:3313200)VisorFSRam: 878: tardiskbackup with (0,115,0,0,1363)      
2021-03-11T19:17:15.405Z cpu3:3313200)FSS: 8524: Mounting fs visorfs (43064c081820) with -o 0,115,0,0,0,755,tardiskbackup on file descriptor 4306a21b9230
2021-03-11T19:17:19.438Z cpu6:3313202)VisorFSTar: 782: nsx_esx_.v00                             
2021-03-11T19:17:19.444Z cpu6:3313202)VisorFSTar: 708: inode 4294970424 (libsnmppluginvs.so) is busy
2021-03-11T19:17:20.455Z cpu2:3313204)VisorFSTar: 782: nsx_esx_.v00                             
2021-03-11T19:17:20.455Z cpu2:3313204)VisorFSTar: 708: inode 4294970424 (libsnmppluginvs.so) is busy                                                                   
2021-03-11T19:17:22.467Z cpu2:3313206)VisorFSTar: 782: nsx_esx_.v00                            
2021-03-11T19:17:22.467Z cpu2:3313206)VisorFSTar: 708: inode 4294970424 (libsnmppluginvs.so) is busy

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

SNMP daemon prevents the NSX-ESX-DATAPATH VIB tardisk from being ummounted

Resolution

Disable SNMP on the ESXi host, before attempting to remove the ESXi Transport Node (TN) from the cluster.