ESXi Host Transport Nodes Disconnected from NSX Manager After Certificate Renewal with error, "Couldn't connect to 'ssl://172.X.X.X:1234' (error: xxxxxxx-certificate verify failed)"

Products

• VMware NSX

Issue/Introduction

Following a successful renewal of the NSX Manager API and cluster certificates, one or more ESXi Host Transport Nodes lose connectivity to the NSX Management Plane and appear as "Disconnected" in the NSX UI.

When administrators attempt to remediate the expired transport node certificates using Alarm For Transport Node Certificate Has Expired - Host/Edges MPA Disconnected , the follow logs are seen in the NSX manager.

To validate the connectivity, you can run the below commands.

Example:

[root@esxi:~] nc -z -w 1 172.X.X.X 1234
Connection to 172.X.X.X1234 port [tcp/*] succeeded!

[root@esxi:~] nc -z -w 1 172.X.X.X 1235
Connection to 172.X.X.X 1235 port [tcp/*] succeeded!

[root@esxi:~] nc -z -w 1 172.X.X.X 443
(Returns null / command times out)

Example log snippets /var/log/nsx-syslog:

In(182) nsx-proxy[...] - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="23064504" level="INFO"] StreamSocket[80 Open f:39 i:0 ? -> ssl://172.X.X.X:1234] on_connect 336134278-certificate verify failed
Wa(182) nsx-proxy[...] - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="23064504" level="WARNING"] StreamConnection[80 Connecting to ssl://172.X.X.X:1234 sid:80] Couldn't connect to 'ssl://172.X.X.X:1234' (error: 336134278-certificate verify failed)

Environment

VMware NSX 4.x

VMware ESXi 8.x

Cause

This issue occurs due to an firewall configuration where an intermediate network firewall explicitly blocks TCP Port 443 (Management Plane) traffic between the ESXi hosts and the NSX Managers, but leaves Control Plane ports (TCP 1234/1235) open.

 

Resolution

To resolve this issue, you must restore connectivity on port 443 so the NSX Manager can authenticate the API calls and push the new certificate thumbprints to the Host Transport Nodes.

• Engage your network security team to allow TCP Port 443 (Bi-directional) between the affected ESXi host VMkernel IPs and the NSX Manager cluster IPs.

Once the firewall connectivity is restored, follow the below steps on NSX Manger.

• Log in to the NSX Manager UI.

• Navigate to System > Fabric > Nodes > Host Transport Nodes.

• Select the checkboxes next to the disconnected hosts.

• Click 'Resolve'. (Alternatively, you can now successfully re-run Step 5 from KB 345825 in the CLI).

The NSX Manager will now successfully push the updated certificate thumbprints over port 443. Once the local trust store is updated, the host will validate the SSL handshake on ports 1234/1235 and return to an "Up" and "Connected" state.

Additional Information

You can definitively prove this asymmetrical firewall block by running the nc (netcat) utility directly from the ESXi host CLI.

If the host is experiencing this issue, the control plane ports will show as succeeded, while port 443 will timeout: