Error in NSX VIBs installation and unable to delete NSX VIBs using "nsxcli -c del nsx" command

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• Trying to remove NSX VIB from ESXi host(s) fails with error message displayed on screen and ESXi log bundle /var/log/esxupdate.log:

2023-04-17T06:44:49Z esxupdate: 3227255: LiveImageInstaller: WARNING: Handling Live Vib Failure: VMware_bootbank_nsx-opsagent_3.0.0.0.0-7.0.15945993: Error in running [/etc/init.d/nsx-opsagent stop upgrade]: Return code: 1 Output: OK to upgrade nsx-opsagent stop nsx-opsagent stop watchdog watchdog-opsAgent[3227723]: Terminating watchdog process with PID 2917842 nsx-opsagent service is stopped cp: can't stat '/etc/vmware/nsxa/host_config.bin': No such file or directory
2023-04-17T06:45:02Z esxupdate: 3227255: HostImage: DEBUG: installer LiveImageInstaller failed: VMware_bootbank_nsx-opsagent_3.0.0.0.0-7.0.15945993: VMware_bootbank_nsx-opsagent_3.0.0.0.0-7.0.15945993: Error in running [/etc/init.d/nsx-opsagent stop upgrade]: Return code: 1 Output: OK to upgrade nsx-opsagent stop nsx-opsagent stop watchdog watchdog-opsAgent[3227723]: Terminating watchdog process with PID 2917842 nsx-opsagent service is stopped cp: can't stat '/etc/vmware/nsxa/host_config.bin': No such file or directory  It is not safe to continue. Please reboot the host immediately to discard the unfinished update.. Clean up the installation.
2023-04-17T06:45:02Z esxupdate: 3227255: vmware.runcommand: INFO: runcommand called with: args = '['/usr/lib/vmware/vob/bin/addvob', 'vob.user.esximage.install.error', "VMware_bootbank_nsx-opsagent_3.0.0.0.0-7.0.15945993: VMware_bootbank_nsx-opsagent_3.0.0.0.0-7.0.15945993: Error in running [/etc/init.d/nsx-opsagent stop upgrade]:\nReturn code: 1\nOutput: OK to upgrade\nnsx-opsagent stop\nnsx-opsagent stop watchdog\nwatchdog-opsAgent[3227723]: Terminating watchdog process with PID 2917842\nnsx-opsagent service is stopped\ncp: can't stat '/etc/vmware/nsxa/host_config.bin': No such file or directory\n\nIt is not safe to continue. Please reboot the host immediately to discard the unfinished update."]', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

• Running 'del nsx' on the ESXi host in the nsxcli shell does not remove the VIB's.
  
  
• In the NSX-T UI removing NSX-T using force, also fails to remove the VIB's.
  
  
• Reboot of the host does not resolve the issue.
  
  
• Issued the "nsxcli -c del nsx" command to remove the NSX VIBs, which returned an error.

% Exception when deleting nsx from host: 'error code: 4 stdout: delete_nsx_instance_from_host.sh: INFO: NSX reset script called with argument fabric_node on nsx-esx delete_nsx_instance_from_host.sh: INFO: Run transport_node reset on ESX node % Failed to remove all host switches or logical switches delete_nsx_instance_from_host.sh: ERROR: Failed to reset nsxa app of nsx-opsagent. Please check ospagent logs for more details.  stderr: ERROR: Failed to reset nsxa app of nsx-opsagent. Please check ospagent logs for more details.'

• nsx-opsagent service is not running. Restarting nsx-opsagent service on the ESXi host using the command "/etc/init.d/nsx-opsagent restart" restarts the service successfully but opsagent status sets back to not running after sometime when validated against command "/etc/init.d/nsx-opsagent status"
  

Environment

VMware NSX-T Data Center 3.x

Cause

VIB deletion (NSX VIBs using "nsxcli -c del nsx" command) is failed because there are some properties which are left to be deleted.

Log location : /var/run/log/nsxaVim.log

2023-04-14T10:17:22Z nsxaVim: [2105560]: INFO Retrieved nsx config [(vim.dvs.HostDistributedVirtualSwitchManager.NsxDvsConfigInfo) {  dynamicType = <unset>,  dynamicProperty = (vmodl.DynamicProperty) [],  globalOpaqueDataList = (vim.dvs.OpaqueData) [],  hostOpaqueDataList = (vim.dvs.OpaqueData) [],  globalKeyValueDataList = (vim.KeyValue) [],  hostKeyValueDataList = (vim.KeyValue) [    (vim.KeyValue) {     dynamicType = <unset>,     dynamicProperty = (vmodl.DynamicProperty) [],     key = 'com.vmware.net.portset.fc.enabled',     value = 'true'    },    (vim.KeyValue) {     dynamicType = <unset>,     dynamicProperty = (vmodl.DynamicProperty) [],     key = 'com.vmware.net.portset.fc.mcast.enabled',     value = 'true'    }  ] }] for dvs [## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##]

2023-04-14T10:17:22Z nsxaVim: [2105560]: INFO All NSX properties are not removed by cfgAgent yet, retry times left 59

Basically when we try to remove the nsx configuration from the host side, we try to remove the nsx properties on the host.

In hostKeyValueDataList, com.vmware.net.portset.fc.enabled and com.vmware.net.portset.fc.mcast.enabled property is not deleted. This is why attempt to delete NSX VIBs using command "nsxcli -c del nsx" fails.

Resolution

To overcome this situation, delete this property manually by using command. Run this command for each property.

Obtain the name of the vDS which needs to be modified via the esxcfg-vswitch -l command.

For each vDS in use by NSX, issue a command similar to the following to disable the active NSX kernel module(s):

Command syntax : net-dvs -u <propertyName> -p hostKeyValueDataList <SwitchName>

Example of commands executed for properties com.vmware.net.portset.fc.enabled and com.vmware.net.portset.fc.mcast.enabled :-

a.) net-dvs -u com.vmware.net.portset.fc.enabled -p hostPropList <SwitchName>

b.) net-dvs -u com.vmware.net.portset.fc.mcast.enabled -p hostPropList <SwitchName>

Steps thereafter :

1. Issue the "nsxcli -c del nsx" command to remove the NSX VIBs.
2. Issue the "esxcli software vib list | grep -E 'nsx|vsipfwlib'" command to validate that no NSX VIBs are installed on the host. There should be no output.
3. If NSX VIB deletion is successful (NSX VIBs no longer listed), configure NSX on the host transport node using NSX UI.
4. If NSX VIB deletion is not successful (NSX VIBs still listed), please engage Broadcom support for further investigation.