Hyperbus Connection Found UNHEALTHY (MISS_VERSION_HANDSHAKE) on a Single VM After NSX T Upgrade to 3.1.3.3

Products

• VMware NSX

Issue/Introduction

Symptoms:

• Some container VM's show a status of 'MISS_VERSION_HANDSHAKE' if queried with the command 'nsxcli -c get hyperbus connection info'.

nsxcli -c get hyperbus connection info

Thu Dec 23 2021 UTC 20:12:31.155

        VIFID              Connection             Status                HostSwitchID

65101bd0-3a0e-4974-aa79-03ba24958a20     169.254.1.10:2345           MISS_VERSION_HANDSHAKE 50 1d 00 40 3f c0 6b 24-a0 ec d8 07 9a 83 48 44

8afe08b9-6f74-4692-a379-7ba903ed44c4     169.254.1.11:2345           HEALTHY  50 1d 00 40 3f c0 6b 24-a0 ec d8 07 9a 83 48 44

• Creation of additional containerized workloads may fail. 
• 'net-stats -l | grep '4094' | wc -l' and compare to the total number of hyperbus connections in 'nsxcli -c get hyperbus connection info' The count should be different by the number of connections with 'status= MISS_VERSION_HANDSHAKE'.

 

Environment

VMware NSX-T Data Center

Cause

The hyperbus pvlans are being removed in the upgrade to NSX-T 3.1.3.3.  A race condition can be hit in rare circumstances that prevents the hyperbus port from being created.   A vMotion or restarting nsx-cfgagent will recreate this port and resolve the issue.

Resolution

Currently, there is no resolution to this behavior.

Workaround:
There are two known workarounds to fix this behavior:

1. vMotion an affected Diego Cell VM to another NSX T 3.1.3.3 host.
OR
2. Restart cfgagent process via SSH access on an affected ESXi host.  This has no impact on the datapath but will correct the problem for affected Diego Cell VM's on the host.
     /etc/init.d/nsx-cfgagent restart