NSX-T ESXi Transport Node upgrade fails

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX-T ESXi Transport Node is upgraded using either in-place or rebootless maintenance upgrade.
• ESXi host logs (esxupdate.log) display message(s) similar to:

#less /var/log/esxupdate.log
Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host 36c9daa1-6a34-423b-a85c-c3ddf3f6eec8 with error : [LiveInstallationError] ([], "Error in running rm /tardisks/nsx_esx_.v00:\nReturn code: 1\nOutput: rm: can't remove '/tardisks/nsx_esx_.v00': Device or resource busy\n\nIt is not safe to continue. Please reboot the host immediately to discard the unfinished update.") Please refer to the log file for more details..

• ESXi host logs (vmkernel.log) display message(s) similar to:

2019-05-15T02:20:17.513Z cpu2:136770)VisorFSTar: 838: inode 5572 (libsnmppluginvs.so) is busy

• SNMP is enabled on the ESXi host:

#esxcli system snmp get

Example of output when SNMP is enabled:
   Authentication:
   Communities:
   Enable: true <<----
   Engineid:
   Hwsrc: indications
   Largestorage: true
   Loglevel: info
   Notraps:
   Port: 161
   (...) output omitted

Environment

VMware NSX-T
VMware NSX-T Data Center 2.x
VMware NSX-T Data Center

Cause

The upgrade fails because a plugin used by SNMP is not stopped which prevents NSX-T ESXi Transport Node in-place upgrade.

Resolution

This issue is resolved in NSX-T 2.4.3 and NSX-T 2.5 (upgrade from those version won't be affected by this issue).

Workaround:
To workaround the issue, disable SNMP before the upgrade and re-enable it after the upgrade is completed using the following commands:
#esxcli system snmp set -e false
#esxcli system snmp set -e true

To verify the SNMP status, the following command can be used:
#/etc/init.d/snmpd status

Expected output when SNMP is running:
snmpd is running

Expected output when SNMP is not running:
snmpd is not running

Alternatively, you can disable SNMP using the vSphere Web client:

Select and ESXi host > click the "Configure" tab > select "Security Profile" > in go to Service and select "Edit..." > find the SNMP service and stop it as shown in the screenshot below: