After NSX 4.x install/upgrade, ESXi Hosts are in a "Not Responding" state in vCenter and VMs cannot connect to their network

Products

• VMware NSX

Issue/Introduction

• ESXi Hosts prepared for NSX are in "Not Responding" state in vCenter.
• Restarting the management agents will not help.
• SSH connection to the ESXi host fails with "Connection refused" even though the service is enabled and and not blocked by any firewall.
• vmkping doesn't work. It returns with an output similar to below

vmkping -I vmk1 -s 8972 -d ###.###.###.###

Socket creation error:socket() returns -1 (No buffer space available)

• esxcli commands do not complete as expected. The command returns an output similar to "No buffer space available" or "Buffer full".

esxcli network nic get -n vmnic0
IO error: [Errno 105] No buffer space available

• VMs may fail to connect to segments or vDS portgroup with the error 'failed to connect virtual device ethernetX'
• ESXi vmkernel logs, /var/log/vmkernel.log, has the error signature "unsupported address family" has entries like below. 

2022-11-15T14:38:50.749Z cpu6:2098499)WARNING: nsxt-ipfix: IpfixNSXSocketCreateOne:51: [nsx@6876 comp="nsx-esx" subcomp="ipfix"]Fail to create socket(domain 1): Unsupported address family

Environment

VMware NSX version below 4.1.0

ESXi hosts have IPv6 disabled.

IPFIX is enabled in the environment.

Cause

The IPFIX module tries to create an IPv6-based socket and fails as IPv6 is disabled, repeated multiple times per second.

Resolution

This issue is resolved in VMware NSX 4.1.0

Workaround:

1. Disable IPFIX in NSX or disable "NetFlow" Monitoring on the PortGroup to which the ESXi host has access to.
2. From DCUI, enable the IPV6 Address for that specific host and Reboot is mandatory.
   1. Login to ESXi Host from DCUI with root.
   2. Select Configure Management Network and press <Enter>
   3. Select IP IPv6 Configuration and press <Enter>
   4. Use arrow keys and select IPV6. If there is no static IPV6 configuration available, select "Use dynamic IPv6 address and network configuration". Select each of the options by entering <SPACE BAR>
   5. Once changes are made and you press <ENTER> you will be returned to the Configure Management Network screen but the changes you just made will not appear until after a reboot.
   6. Hit the <Esc> key and you will see a confirmation screen. Press <Y> to continue. Press <N> to return the network configuration to its original state, or <Esc> to continue making changes.
   7. After pressing <Y> the host will reboot.
3. If the Configure Management Network option is grayed out in the DCUI console, then access the CLI (command prompt) and run the following command to enable the IPv6 TCP/IP stack:
   esxcli network ip set --ipv6-enabled=true

Additional Information

Impact/Risks:
In this stage, it is not possible to enter a host in maintenance mode to ensure accessibility for vSAN objects and its VMs, nor to evacuate the VMs off that host.