ESXi transport node may face PSOD during LACP port creation with ENS enabled on its N-VDS

Products

• VMware NSX

Issue/Introduction

Symptoms:

You can see the backtrace related to ENS on vmkernel core with PF Exception type 14. The backtrace may be different each time it occurs because it depends on page fault timing. LACP port creation is operated during edit lag configuration or NSX-T upgrade with LACP configuration.

Environment

VMware NSX-T Data Center

Cause

With ENS enabled, ESXi Host cannot handle large LACP port index properly which is managed in inside operation. If LACP port is created with over 127 port index during LACP configuration, it leads to pointer address corruption. ESXi Host will face PSOD when it access to the pointer address because it is not loaded on physical memory.
If LACP port creation is operated with all of the following conditions (a)-(c) met, you will face this PSOD.

(a) ESXi version is prior to 7.0

(b) ENS is enabled on the N-VDS

(c) Current largest port index is larger than 122 on the N-VDS

Resolution

This issue is fixed in ESXi 7.0.

Workaround:
If current port index is larger than 122, reboot ESXi Host to reset port index for workaround.
You can check current largest port index as follows.
 

(1) Power on a VM whose vnic is connected to the N-VDS just before LACP port creation. The largest port index will be assigned to its VM's vnic.

(2) Check port indices. In this example, largest port index is 125. ESXi Host reboot is needed to avoid this PSOD before LACP port creation.

# ps=DvsPortset-0
# for num in `vsish -e ls /net/portsets/${ps}/ports`; do vsish -e cat /net/portsets/${ps}/ports/${num}status | egrep "port index|clientName"; done

   port index:1
   clientName:Management
   port index:2
   clientName:LACP_MgmtPort
   port index:3
   clientName:lag
   port index:4
   clientName:vmnic1
   port index:5
   clientName:Shadow of vmnic1
   port index:6
   clientName:vmnic2
   port index:7
   clientName:Shadow of vmnic2
   port index:9
   clientName:L2CtrlPort
   port index:10
   clientName:vdr-vdrPort
   port index:125
   clientName:vm-test.eth0