After NSX-T upgrade to 2.5, the hyperBus interface vmk50 is not present on ESXi hosts and Hyperbus status is MISS_VERSION_HANDSHAKE

Products

• VMware NSX

Issue/Introduction

Purpose is to overcome container creation failure due to HyperBus interface vmk50 not being present on ESXI.

Symptoms:

• Running the following command shows vmk50 missing: 

# esxcfg-vmknic -l
Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack
vmk0 132 IPv4 10.244.64.34 255.255.255.0 10.244.64.255 ##:##:##:##:##:7e 1500 65535 true STATIC defaultTcpipStack
vmk0 132 IPv6 fe80::2083:2fff:fe60:7e 64 ##:##:##:##:##:7e 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk1 110 IPv4 10.0.0.34 255.255.255.0 10.0.0.255 ##:##:##:##:##:3d 1500 65535 true STATIC defaultTcpipStack
vmk1 110 IPv6 fe80::250:56ff:fe67:393d 64 ##:##:##:##:##:3d 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk2 82 IPv4 44.162.64.34 255.255.128.0 44.162.127.255 ##:##:##:##:##:c1 1500 65535 true STATIC defaultTcpipStack
vmk2 82 IPv6 fe80::250:56ff:fe6d:72c1 64 ##:##:##:##:##:c1 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk10 10 IPv4 192.168.100.54 255.255.255.0 192.168.100.255 ##:##:##:##:##:ec 1600 65535 true STATIC vxlan
vmk10 10 IPv6 fe80::250:56ff:fe65:5eec 64 ##:##:##:##:##:ec 1600 65535 true STATIC, PREFERRED vxlan
[root@tec4-vm-ki1-03:~]

• Running the following command shows hyperbus status as "MISS_VERSION_HANDSHAKE"

# nsxcli -c get hyperbus connection info
                VIFID                    Connection          Status

########-####-####-####-########0a21 169.254.1.53:2345 MISS_VERSION_HANDSHAKE

• vmkping to container hyper bus will show host down.

vmkping ++netstack=hyperbus -I vmk50 169.254.1.53 shows host is down.

• EXi nsx-syslog shows the following hyperbus interface deletion event:

33118 <182>1 2019-10-22T15:23:08.249Z hostname.exmaple.com NSX 95573 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="95822" level="INF O"] [ApplyVTEP] Invoke command to delete HyperBus interface on hostswitch(b6 b7 11 fc ## ## ## ##-## ## ## ## e2 2e 24 6b) reply: [ok]

• ESXi log nsxaVim.log shows it tried to delete vmk50 and the related dvport, here vmk50 is removed but the dvport failed to be deleted.

2019-10-22T15:23:07Z nsxaVim: [95694]: INFO HyperBus interface config delete received^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO Removing HyperBusInterface...^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO vxlan netStackInstance defaultGateway=None.^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO zoneUuid2dvsUuidMap = {'########-####-####-####-########c70f': 'b6 b7 11 fc ## ## ## ##-## ## ## ## e2 2e 24 6b'
}^@
2019-10-22T15:23:07Z nsxaVim: [95694]: INFO Updating vmknicToSpec map on VTEP update^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO vxlan netStackInstance defaultGateway=None.^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO zoneUuid2dvsUuidMap = {'########-####-####-####-########c70f': 'b6 b7 11 fc ## ## ## ##-## ## ## ## e2 2e 24 6b'
}^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO Updating vmknicToSpec map on VTEP update^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO Removed [vmk50] successfully^@
2019-10-22T15:23:08Z nsxaVim: [95694]: INFO Deleting HyperBusInterface port(, ########-####-####-####-########dd37)...^@

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x

Cause

This issue occurs due to the stale dvport for vmk50 which was not removed and lead to failed recreation of vmk50 (hyperbus) again.

If we deploy a new Container, it is not created because the HyperBus interface vmk50 is missing on vSphere ESXi.
For existing Containers , the communication over hyperbus will be completely down.

Resolution

This issue is resolved in NSX-T 2.5.1.

Workaround:
1.Retrieve the vmk50 port ID  and SwitchName using CLI on vSphere ESXi
net-dvs | grep vmk50 -C 10
net-stats -l
EXAMPLE:-

2.Create the vmk50 interface on vSphere ESXi.
esxcli network ip interface add -P <port-id from step-1> -s DvsPortset-0 -i vmk50 -N hyperbus
EXAMPLE:-

3.Assign an IP address to the vmk50 interface.
esxcfg-vmknic -i 169.254.1.1 -n 255.255.0.0 -s DvsPortset-0 -v <port-id from step-1> -N hyperbus
EXAMPLE:-

4. Verify if vmk50 correctly listed:-