After NSX-T upgrade to 2.5, the hyperBus interface vmk50 is not present on ESXi hosts and Hyperbus status is MISS_VERSION_HANDSHAKE

Products

• VMware NSX Networking

Issue/Introduction

Purpose is to overcome container creation failure due to HyperBus interface vmk50 not being present on ESXI.

Symptoms:

• Running the following command shows vmk50 missing: 

# esxcfg-vmknic -l
Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack
vmk0 132 IPv4 10.244.64.34 255.255.255.0 10.244.64.255 22:83:2f:60:00:7e 1500 65535 true STATIC defaultTcpipStack
vmk0 132 IPv6 fe80::2083:2fff:fe60:7e 64 22:83:2f:60:00:7e 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk1 110 IPv4 10.0.0.34 255.255.255.0 10.0.0.255 00:50:56:67:39:3d 1500 65535 true STATIC defaultTcpipStack
vmk1 110 IPv6 fe80::250:56ff:fe67:393d 64 00:50:56:67:39:3d 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk2 82 IPv4 44.162.64.34 255.255.128.0 44.162.127.255 00:50:56:6d:72:c1 1500 65535 true STATIC defaultTcpipStack
vmk2 82 IPv6 fe80::250:56ff:fe6d:72c1 64 00:50:56:6d:72:c1 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk10 10 IPv4 192.168.100.54 255.255.255.0 192.168.100.255 00:50:56:65:5e:ec 1600 65535 true STATIC vxlan
vmk10 10 IPv6 fe80::250:56ff:fe65:5eec 64 00:50:56:65:5e:ec 1600 65535 true STATIC, PREFERRED vxlan
[root@tec4-vm-ki1-03:~]

• Running the following command shows hyperbus status as "MISS_VERSION_HANDSHAKE"

# nsxcli -c get hyperbus connection info
                VIFID                    Connection          Status

b34f0f56-1220-4ede-87b7-8c9604ee0a21 169.254.1.53:2345 MISS_VERSION_HANDSHAKE

• vmkping to container hyper bus will show host down.

vmkping ++netstack=hyperbus -I vmk50 169.254.1.53 shows host is down.

• EXi nsx-syslog shows the following hyperbus interface deletion event:

33118 <182>1 2019-10-22T15:23:08.249Z tec4-vm-ki1-03.teco.local NSX 95573 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="95822" level="INF O"] [ApplyVTEP] Invoke command to delete HyperBus interface on hostswitch(b6 b7 11 fc 08 46 49 71-9b 16 3b c3 e2 2e 24 6b) reply: [ok]

• ESXi log nsxaVim.log shows it tried to delete vmk50 and the related dvport, here vmk50 is removed but the dvport failed to be deleted.

2019-10-22T15:23:07Z nsxaVim: [95694]: INFO HyperBus interface config delete received^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO Removing HyperBusInterface...^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO vxlan netStackInstance defaultGateway=None.^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO zoneUuid2dvsUuidMap = {'31fd3b63-956e-4cae-b0fa-d09a9956c70f': 'b6 b7 11 fc 08 46 49 71-9b 16 3b c3 e2 2e 24 6b'
}^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO Updating vmknicToSpec map on VTEP update^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO vxlan netStackInstance defaultGateway=None.^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO zoneUuid2dvsUuidMap = {'31fd3b63-956e-4cae-b0fa-d09a9956c70f': 'b6 b7 11 fc 08 46 49 71-9b 16 3b c3 e2 2e 24 6b'
}^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO Updating vmknicToSpec map on VTEP update^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO Removed [vmk50] successfully^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO Deleting HyperBusInterface port(, 8c06908e-ec36-4f36-973e-062ea7ffdd37)...^@

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x

Cause

This issue occurs due to the stale dvport for vmk50 which was not removed and lead to failed recreation of vmk50 (hyperbus) again.

If we deploy a new Container, it is not created because the HyperBus interface vmk50 is missing on vSphere ESXi.
For existing Containers , the communication over hyperbus will be completely down.

Resolution

This issue is resolved in NSX-T 2.5.1.

Workaround:
1.Retrieve the vmk50 port ID  and SwitchName using CLI on vSphere ESXi
net-dvs | grep vmk50 -C 10
net-stats -l
EXAMPLE:-

2.Create the vmk50 interface on vSphere ESXi.
esxcli network ip interface add -P <port-id from step-1> -s DvsPortset-0 -i vmk50 -N hyperbus
EXAMPLE:-

3.Assign an IP address to the vmk50 interface.
esxcfg-vmknic -i 169.254.1.1 -n 255.255.0.0 -s DvsPortset-0 -v <port-id from step-1> -N hyperbus
EXAMPLE:-

4. Verify if vmk50 correctly listed:-