Extra large edge node fails to boot up properly after EVC configuration

Products

• VMware NSX

Issue/Introduction

• EVC mode is configured on cluster with CPU below Haswell generation.
• Extra large edge nodes do not boot up properly.
• NSX Manager UI shows Edge interfaces eth1, ethe2 and eth3 in admin down state.
• The fp-eth# interfaces are not present.
• The datapath service does not start.
  
  Edge##-01> get service dataplane
Thu Nov 17 2022 UTC 22:39:50.156
Service name: dataplane
Service state: stopped

• Small edge nodes do not show this issue.

 

Environment

VMware NSX

Cause

In logs and vmx file show that the PDPE1GB requirement is not satisfied. This results in the Edge node failing to power on.
This will be due to the EVC mode blocking the 1G support from guest VM.

In vmfs/volumes/<path>/<VM-name>/vmware.log the capability PDPE1G is missing

2022-12-14T05:49:50.521Z| vmx| I125: FeatureCompat: Capabilities:
2022-12-14T05:49:50.521Z| vmx| I125: Capability Found: cpuid.Intel = 0x1

<etc, no PDPE1GB capability visible in logs>

2022-12-14T05:49:50.521Z| vmx| I125: Capability Found: cpuid.IBRS = 0x1

2022-12-14T05:49:50.519Z| vmx| I125: Full guest CPUID with differences from hostCPUID highlighted.
...
2022-12-14T05:49:50.519Z| vmx| I125: guest vs. host CPUID guest codename: Sandy Bridge EP
2022-12-14T05:49:50.519Z| vmx| I125: guest vs. host CPUID *host codename: Broadwell EP/EN/EX

More than 16GB of datapath hugepage memory is required when using 2MB hugepages. This is why extra large edge node does not function while small edge nodes have no problem.

Resolution

This is an expected behavior. Use EVC mode Haswell or later to get 1G hugepage support.

Reference: NSX Edge VM System Requirements

If vSphere EVC mode is enabled for the host for the NSX Edge VM, the CPU must be Haswell or later generation.

Workaround:

• Set hugepage to 16G or lower value.
edge> set hugepage-reserve size 16384MB
• Reboot the VM.