After NSX 4.x install/upgrade, ESXi Hosts are in a "Not Responding" state in vCenter and VMs cannot connect to their network

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX 4.0.0.1/4.0.1.1
• ESXi hosts have IPv6 disabled.
• IPFIX is enabled in the environment.
• ESXi Hosts prepared for NSX are in "Not Responding" state in vCenter.
• Restarting the management agents will not help.
• SSH connection to the ESXi host fails with "Connection refused" even though the service is enabled and and not blocked by any firewall.
• vmkping doesn't work. It returns with an output similar to below

[root@esxihost:~] vmkping -I vmk1 -s 8972 -d 192.168.100.x

Socket creation error:socket() returns -1 (No buffer space available)

• esxcli commands do not complete as expected. The command returns an output similar to "No buffer space available" or "Buffer full".

[root@esxihost:~] esxcli network nic get -n vmnic0
IO error: [Errno 105] No buffer space available

• VMs may fail to connect to segments or vDS portgroup with the error 'failed to connect virtual device ethernetX'
• ESXi vmkernel logs, /var/log/vmkernel.log, has the error signature "unsupported address family" has entries like below. 

2022-11-15T14:38:50.749Z cpu6:2098499)WARNING: nsxt-ipfix: IpfixNSXSocketCreateOne:51: [nsx@6876 comp="nsx-esx" subcomp="ipfix"]Fail to create socket(domain 1): Unsupported address family

Environment

VMware NSX-T Data Center 4.x
VMware NSX-T Data Center

Cause

The ipfix module tries to create an IPv6-based socket and fails as IPv6 is disabled, repeated multiple times per second.

Resolution

This issue is resolved in VMware NSX 4.1.0 available at VMware Downloads.

Workaround:
1. Disable IPFIX in NSX-T or disable "NetFlow" Monitoring on the PortGroup to which the ESXi host has access to.
2. From DCUI Enable the IPV6 Address for that specific host and Reboot is mandatory.

• Login to ESXi Host from DCUI with root.
• Select Configure Management Network and press <Enter>
• Select IP IPv6 Configuration and press <Enter>
• Use arrow keys and select IPV6 and if the customer doesn't have static IPV6,  then select "Use dynamic IPv6 address and network configuration". Select each of the options by entering <SPACE BAR>
• Once your changes are made and you press <ENTER> you will be returned to the Configure Management Network screen but the changes you just made will not appear until after a reboot.
• Hit the <Esc> key and you will see a confirmation screen. Press <Y> to continue. Press <N> to return the network configuration to its original state, or <Esc> to continue making changes.
• After pressing <Y> the host will reboot.

Additional Information

Impact/Risks:
In this stage, it is not possible to enter a host in maintenance mode to ensure accessibility for vSAN objects and its VMs, nor to evacuate the VMs off that host.