Host transport node in Partial Success after update of 3rd party VIBs

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

• NSX-T Data Center (any version)
• NSX (any version)
• After installation or patching of 3rd party VIBs, host transport node (HTN) may go to "Partial Success" state.
• NSX Manager may show error similar to:
  "Host configuration: host switch apply operation failed; [socket read failed]; socket read failed;"
• vMotion to these HTNs may fail with
  "A general system error occurred: Invalid fault". 
• Command "esxcli software vib list" will contain logs similar to sample below:
  esxcli software vib list | head -n 2
Error loading plugin /usr/lib/vmware/esxcli/ext/<plugin-name>.xml skipping.  Error was: Error while trying to register the plugin /usr/lib/vmware/esxcli/ext/<plugin-name>.xml Unable to find a command or namespace with the name '<name>' in namespace <plugin-name> controller The functionality in /usr/lib/vmware/esxcli/ext/<plugin-name>.xml will not be available until this issue is resolved.
Error loading plugin /usr/lib/vmware/esxcli/ext/<plugin-name>.xml skipping.  Error was: Error while trying to register the plugin /usr/lib/vmware/esxcli/ext/<plugin-name>.xml Unable to find a command or namespace with the name '<name>' in namespace <plugin-name> controller The functionality in /usr/lib/vmware/esxcli/ext/<plugin-name>.xml will not be available until this issue is resolved.
• /var/run/log/nsxaVim.log may throw an exception similar to the sample below:
  2024-05-28T15:50:45Z Er(11)[+] nsxaVim: Traceback (most recent call last):
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: File "/usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py", line 4877, in Server
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: result = ProcessCommand(data.decode('utf-8'))
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: File "/usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py", line 4585, in ProcessCommand
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: return FetchSmartNICEnabledDvsInfo()
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: File "/usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py", line 1551, in FetchSmartNICEnabledDvsInfo
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: version, update, patch = getEsxiVersion()
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: File "/usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/esxVersionUtil.py", line 108, in getEsxiVersion
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: raise Exception('Failed to get ESXi version: %s - %s' % (out, e))
2024-05-28T15:50:45Z Er(11)[+] nsxaVim: Exception: Failed to get ESXi version: Error loading plugin /usr/lib/vmware/esxcli/ext/<plugin_name>.xml skipping. Error was: Error while trying to register the plugin /usr/lib/vmware/esxcli/ext/<plugin_name>.xml Unable to find a command or namespace with the name '<name>' in namespace storcli controller The functionality in /usr/lib/vmware/esxcli/ext/<plugin_name>.xml will not be available until this issue is resolved.
• Similar logs will be observed in host's:
  /var/run/log/nsx-syslog
/var/run/log/syslog

Environment

VMware NSX-T Data Center
VMware NSX

Cause

Realization of Transport Node will fail due to failing call to validate ESXi version.

This call is routinely executed during host transport node realization, when the issue is present, it will fail with exception. 

Cause of this issue is a corrupt, or incorrectly packaged 3rd party VIB. 

Resolution

This is a known issue, which will be fixed in the future.

To avoid such behavior, please make sure the VIBs installed on the ESXi host are healthy.