After NSX 4.x upgrade, Edge node status is down together with BGP and TEP tunnels

Products

• VMware NSX

Issue/Introduction

• You upgraded to or installed VMware NSX 4.x version below 4.1.0
• Edge Node Status is down
• BGP is down
• Tunnels are not available or down
• ESXi Hosts with the affected Edge VMs on them are prepared for NSX
• ESXi hosts may or may not be in "Not Responding" state in vCenter.
• ESXi hosts have IPv6 disabled.
• IPFIX is enabled in the environment.
• On the edge the below error is seen after issuing command: 
  get logical-routers
% An unexpected error occurred: The dataplane service is in error state, has failed or is disabled

  When entering VRF 0 which is used by the TEP tunnels, you see the error:
  vrf 0
% An internal error occurred

• Core dumps are present for core.dp-ipc and were generated after the upgrade in /var/log/core/
-rw-r--r--  1 root root  14M Jun 18 16:01 core.dp-ipc30.#########.#####.#.#.gz
-rw-r--r--  1 root root 249M Jun 18 16:04 core.dp-ipc31.#########.#####.#.#.gz
-rw-r--r--  1 root root 205M Jun 19 10:49 core.dp-ipc31.#########.#####.#.#.gz

Environment

VMware NSX version below 4.1.0

Cause

The IPFIX module tries to create an IPv6-based socket and fails as IPv6 is disabled, repeated multiple times per second.

Resolution

This issue is resolved in VMware NSX 4.1.0

Workaround:

1. Disable IPFIX in NSX or disable "NetFlow" Monitoring on the PortGroup to which the ESXi host has access to.
2. From DCUI, enable the IPV6 Address for that specific host and Reboot is mandatory.
   1. Login to ESXi Host from DCUI with root.
   2. Select Configure Management Network and press <Enter>
   3. Select IP IPv6 Configuration and press <Enter>
   4. Use arrow keys and select IPV6. If there is no static IPV6 configuration available, select "Use dynamic IPv6 address and network configuration". Select each of the options by entering <SPACE BAR>
   5. Once changes are made and you press <ENTER> you will be returned to the Configure Management Network screen but the changes you just made will not appear until after a reboot.
   6. Hit the <Esc> key and you will see a confirmation screen. Press <Y> to continue. Press <N> to return the network configuration to its original state, or <Esc> to continue making changes.
   7. After pressing <Y> the host will reboot.
3. If the Configure Management Network option is grayed out in the DCUI console, then access the CLI (command prompt) and run the following command to enable the IPv6 TCP/IP stack:
   esxcli network ip set --ipv6-enabled=true

Additional Information

After NSX 4.x install/upgrade, ESXi Hosts are in a "Not Responding" state in vCenter and VMs cannot connect to their network