NSX DFW Policy/Rule status is unknown due to duplicate host entry

Products

• VMware vDefend Firewall
• VMware NSX

Issue/Introduction

• In the NSX UI, under Security - Distributed Firewall, the status of a published policy or rule is 'Unknown'.

Clicking on the 'Unknown' status, displays the policy or rule status for the relevant host(s).

You observe a duplicate entry (one host is listed twice).  One entry has a status of 'Success' and the other has a status of 'Unknown':

• Run the following command from NSX Manager (admin mode) to list all transport nodes with the relevant host(s) listed twice with different UUID's:

> get nodes
UUID                    Type  Display Name
<Transport-Node-UUID-1>   esx   esx-01.example.com
<Transport-Node-UUID-2>   esx   esx-01.example.com

• Run run the following API call from NSX Manager (root mode) for each UUID discovered in the previous command.  One UUID displays status 'Up'and the other UUID displays status 'Unknown':

# curl -k -u 'admin' -H "Content-Type: application/xml" -X GET https://localhost/api/v1/transport-nodes/<Transport-Node-UUID>/status

Result for Transport-Node-UUID-1:

"node_uuid" : "<Transport-Node-UUID-1>",
"node_display_name" : "esx-01.example.com",
"status" : "UP",
...
"status" : "UP",

Result for Transport-Node-UUID-2:

"node_uuid" : "<Transport-Node-UUID-2>",
"node_display_name" : "esx-01.example.com",
"status" : "UNKNOWN",
...
"status" : "UNKNOWN",

• Attempitng to delete the host with a status of Unknow via the below API call results in the following error:

# curl -k -u 'admin' -H "Content-Type: application/xml" -X DELETE https://localhost/api/v1/transport-nodes/<Transport-Node-UUID>

 

"httpStatus" : "BAD_REQUEST",
"error_code" : 9411,
"module_name" : "NsxSwitching service",
"error_message" : "Cannot delete a transport node <Transport-Node-UUID> which is part of Auto-TN compute collection f9412a40-####-####-####-d63d21c9d0e0:domain-c##. Please use the object name or UUID in Global Search to find all linked objects."​​​​​

• In the NSX Manager logs:

/var/log/syslog*

2023-03-09T10:52:16.138+01:00 nsxmanager01 NSX 6075 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" reqId="16a63b47-####-####-####-28938ab21484" subcomp="manager" username="####@example.com"] MPA on <Inactive-Transport-Node-UUID> is not connected.
...
2023-03-09T10:52:16.143+01:00 nsxmanager01 NSX 6075 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" reqId="16a63b47-####-####-####-28938ab21484" subcomp="manager" username="####@example.com"] Got deployment status HOST_DISCONNECTED for node <Inactive-Transport-Node-UUID>

Environment

VMware NSX-T Data Center 3.x

VMware NSX 4.x

Cause

This issue occurs when previously removed hosts were not cleanly uninstalled, leaving stale entries in NSX.

Resolution

This is a known issue impacting NSX.

Workaround

IMPORTANT
The below workaround is not applicable if the cluster where the 'duplicate host' resides is prepared using vSphere Lifecycle Management (vLCM) or has service insertion deployed or is an NSX Security Only installed cluster.  It is not possible to detach the transport node profile (TNP) from such clusters.

If your NSX configuration is not compatible with the below workaround please contact VMware Technical Support and reference this KB article.

1. Detach the transport node profile from the cluster with the 'duplicate host' :
   
   • In NSX UI, under System - Fabric - Hosts, click on the 'Clusters' tab.
   • Select the checkbox for the cluster we want to detach the TNP from.
   • Click Actions - Detach Transport Node Profile
     
     
     
     
2. As root, use the following Delete API call to remove the Unknown/Duplicate host:
   
   # curl -k -u 'admin' -H "Content-Type: application/xml" -X DELETE https://localhost/api/v1/transport-nodes/<Transport-Node-UUID>

     3. Re-apply the TNP to the cluster:

• • In NSX UI under System - Fabric - Hosts, click on the 'Clusters' tab
  • Select the checkbox next to the cluster.
  • Click Configure NSX and select the transport node profile removed in Step 1 above.

    4. Confirm in the DFW that the status of the published policy or rule now shows 'Success' and that only a single host entry exists for the 'duplicate host':

• • In NSX UI under Security - Distributed Firewall 
  • Clicking 'Success' shows that the 'duplicate host' now only has a single entry and the status of this entry is 'Success''.