"nsxcli -c del nsx" command fails to remove NSX VIBs due to nsx-opsagent failing to stop properly

Products

• VMware NSX

Issue/Introduction

• Even after running the 'nsxcli -c del nsx' command, you can confirm that VIBs still remain on the target ESXi host by running the esxcli software vib list | grep -E 'nsx|vsipfwlib' command.
• The following logs are recorded in /var/run/log/esxupdate.log:
  

  <Timestamp> Db(15) esxupdate[2119588]: Output: opsAgent stop watchdog
  <Timestamp> Db(15)[+] esxupdate[<pid>]: watchdog-opsAgent[<pid>]: Terminating watchdog process with PID <pid>
  <Timestamp> Db(15)[+] esxupdate[<pid>]: sh: you need to specify whom to kill
  ...
  <Timestamp> Er(11) esxupdate[<pid>]: Output: rm: can't remove '/tardisks/nsx_opsa.v00': Device or resource busy
  <Timestamp> Er(11) esxupdate[<pid>]: During handling of the above exception, another exception occurred:
  <Timestamp> Er(11) esxupdate[<pid>]: Traceback (most recent call last):
  <Timestamp> Er(11) esxupdate[<pid>]: File "/usr/lib/vmware/esxcli-software", line 1135, in <module>
  <Timestamp> Er(11) esxupdate[<pid>]: main()
  <Timestamp> Er(11) esxupdate[<pid>]: File "/usr/lib/vmware/esxcli-software", line 1122, in main
  <Timestamp> Er(11) esxupdate[<pid>]: ret = CMDTABLE[command](options)
  <Timestamp> Er(11) esxupdate[<pid>]: File "/usr/lib/vmware/esxcli-software", line 714, in ComponentRemoveCmd
  <Timestamp> Er(11) esxupdate[<pid>]: res = t.RemoveComponents(nameids,
  <Timestamp> Er(11) esxupdate[<pid>]: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 564, in RemoveComponents
  <Timestamp> Er(11) esxupdate[<pid>]: self._validateAndInstallProfile(newProfile,
  <Timestamp> Er(11) esxupdate[<pid>]: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1402, in _validateAndInstallProfile
  <Timestamp> Er(11) esxupdate[<pid>]: self.host.Remediate(newprofile, checkmaintmode,
  <Timestamp> Er(11) esxupdate[<pid>]: File "/lib64/python3.8/site-packages/vmware/esximage/HostImage.py", line 1328, in Remediate
  <Timestamp> Er(11) esxupdate[<pid>]: installer.Remediate(checkmaintmode=checkmaintmode,
  <Timestamp> Er(11) esxupdate[<pid>]: File "/lib64/python3.8/site-packages/vmware/esximage/Installer/LiveImageInstaller.py", line 359, in Remediate
  <Timestamp> Er(11) esxupdate[<pid>]: self.liveimage.Remediate(adds, removes, checkmaintmode,
  <Timestamp> Er(11) esxupdate[<pid>]: File "/lib64/python3.8/site-packages/vmware/esximage/Installer/InstallerCommon.py", line 985, in Remediate
  <Timestamp> Er(11) esxupdate[<pid>]: raise Errors.LiveInstallationError(cause, vibs, msg)
  <Timestamp> Er(11) esxupdate[<pid>]: vmware.esximage.Errors.LiveInstallationError: VMware_bootbank_nsx-opsagent_4.1.2.3.0-8.0.23382415: VMware_bootbank_nsx-opsagent_4.1.2.3.0-8.0.23382415: Failed to unmount tardisk nsx_opsa.v00 of VIB VMware_bootbank_nsx-opsagent_4.1.2.3.0-8.0.23382415: Error in running [/bin/rm /tardisks/nsx_opsa.v00]:
  <Timestamp> Er(11) esxupdate[<pid>]: Return code: 1
  <Timestamp> Er(11) esxupdate[<pid>]: Output: rm: can't remove '/tardisks/nsx_opsa.v00': Device or resource busy

• The following logs are recorded in /var/run/log/vmkernel.log:
  

  <Timestamp> In(182) vmkernel: cpu0:<pid>)VisorFSTar: 1101: nsx_opsa.v00
  <Timestamp> In(182) vmkernel: cpu0:<pid>)VisorFSTar: 873: inode <inode> (opsAgent) is busy
  <Timestamp> In(182) vmkernel: cpu75:<pid>)VisorFSTar: 1101: nsx_opsa.v00
  <Timestamp> In(182) vmkernel: cpu75:<pid>)VisorFSTar: 873: inode <inode> (opsAgent) is busy
  <Timestamp> In(182) vmkernel: cpu85:<pid>)VisorFSTar: 1101: nsx_opsa.v00
  <Timestamp> In(182) vmkernel: cpu85:<pid>)VisorFSTar: 873: inode <inode> (opsAgent) is busy

Environment

VMware NSX

Cause

This issue occurs because the nsx-opsagent process fails to terminate properly during the stop routine, leaving the process running.

Resolution

This issue is resolved in VMware NSX 4.2.0.

If you encounter this issue, please follow the steps.

1. Reboot the ESXi host.
2. Retry running the "nsxcli -c del nsx" command.