After ESX host reboot, the Transport Node status is partially successful and won't succeed

Products

• VMware NSX

Issue/Introduction

• In rare cases after an ESXi host reboot, the host shows as "Partial Success" in the NSX UI under Fabric -> Host -> Node
  
• You see messages similar to the following in /var/run/log/nsx-syslog on the ESXi host:
  
  2024-03-13T07:13:55.427Z nsx-opsagent[2102926]: NSX 2102926 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="2103675" level="WARNING"] VDRVMK_GetVMAC(<host switch UUID>) failed with code -1


• Example output in the desired_state_manager.json file in an NSX Manager log bundle, under section  /nsxapi/api/v1/transport-nodes/state:
  
   {
     "details": [
       {
          "failure_code": 8804,
          "failure_message": " Host configuration: VDRVMK_GetVMAC(<host switch UUID>) failed with code -1; LogicalSwitch full-sync: LogicalSwitch full-sync realization query skipped.",
          "state": "partial_success",
          "sub_system_id": "<UUID>",
           "sub_system_type": "Host"
        }
  ],

Environment

VMware NSX

Cause

After an ESXi host reboot, the nsxa app restarts and reconnects with the VDR. Part of the appInit process checks if the VDR MAC addresses have changed. In a rare occurrence, the VDRVMK_GetVMAC can return with errno = ENOENT which is treated as a failure and causes the host to enter the Partial Success state.

Resolution

This issue is resolved in VMware NSX 3.2.4
This issue is resolved in VMware NSX 4.2.0

Workaround:

1. Restart nsx agent on the affected ESXi:
   
   On NSX 3.2 or later releases:
   
   /etc/init.d/nsx-opsagent stop
/etc/init.d/nsx-opsagent start
   
   For verification if the nsx-opsagent is running, please use the below command
   
   /etc/init.d/nsx-opsagent status

   

   On NSX 3.1 and previous releases:

   /etc/init.d/nsxa stop
/etc/init.d/nsxa start

   For verification if the nsx-opsagent is running, please use the below command

   /etc/init.d/nsxa status


2. You can alternatively reboot the affected host.

The probability of re-occurrence is extremely low.

Additional Information

Impact/Risks:
If the ESXi host is taken out of Maintenance Mode in the Partial Success state, and VMs are then vMotioned to this host, those VMs will lose connectivity with machines on other networks.

Perform below checks if multiple TNs remain in Partial Success state :

Check the managers and controllers connection from Transport nodes using below command  

>> nsxcli -c get managers  (all 3 managers should stay connected) 

>> nsxcli -c get controllers (one of the NSX manager/controller should stats connected status)

Check the open port between NSX managers and the ESXi hosts using below commands

>> nc -zv <NSX manager IP> 1234 (for manager connectivity)

>> nc -zv <NSX manager IP> 1235 (for controller connectivity)

If any of the managers are not in connected state and port is not reachable from multiple hosts, then reboot that NSX manager or power OFF/ON.

Once NSX manager comes back online, make the previous checks and ensure connectivity is restored with multiple Transport nodes.