NSX-T 4.2.4 Manager/Edge appliances fail to install or upgrade due to missing RDSEED support on unsupported CPU or incompatible EVC configurations

Products

VMware NSX

Issue/Introduction

During new installations or upgrades to version 4.2.4, NSX-T appliances (NSX Manager and Edge) may get stuck in a reboot. This issue occurs if the vSphere cluster or VM level EVC baseline is configured incorrectly, or if the host uses an unsupported CPU series based on vSphere 8.0.x compatibility requirements or if the RDSEED instruction set is not available to the NSX appliance from the underlying hardware.
NSX-T 4.2.4 Manager boot after fresh install or during upgrade gets stuck with console showing:

NSXT-4.2.4 Edge boot after fresh install gets stuck with console showing:

Edge Upgrade to NSX-T 4.2.4 fails with below error on NSX UI and API:
Edge 4.2.4.0.0.25410638/Edge/nub/VMware-NSX-edge-4.2.4.0.0.25410643.nub post power on task failed on edge TransportNode dff26e79-###-####-8765-d9343417db89: clientType EDGE , target edge fabric node id dff26e79-####-####-8765-d9343417db89, return status Issuing post_power_on failed to complete in time.
From the Upgrade UI in NSX, the failure of the Edge upgrade may also report as:
Upgrade Agent on Edge node dff26e79-####-####-8765-d9343417db89 is unreachable. Restart the Upgrade agent service and check network connectivity.
User is unable to login using admin/audit user, root login might work.

Environment

NSX 4.2.4

This issue impacts vSphere clusters hosting NSX-T appliances under the following conditions:

1. Unsupported VMware EVC Baseline Configurations

Intel CPU Mode: Intel "Haswell" Generation (L6) or earlier.
AMD CPU Mode: AMD Opteron™ "Steamroller" Generation (B3) or earlier.

2. Unsupported Server CPU Series (with EVC Disabled)

Physical servers running

Intel "Haswell" (or earlier) processors.
AMD Opteron™ "Steamroller" (or earlier) processors.

3. NSX Appliance does not have RDSEED instruction set support available.

This can be confirmed by running the following command on the NSX appliance:

cat /proc/cpuinfo | grep rdseed

Cause

NSX 4.2.4 appliances require the RDSEED CPU instruction set to function. If this feature is missing from the underlying physical server's hardware or is masked by a legacy vSphere Enhanced vMotion Capability (EVC) baseline, the appliance cannot boot properly.

Resolution

This is a known behaviour impacting VMware NSX 4.2.4.

To permanently resolve this issue, perform the following actions based on your environment:

NSX Manager and NSX Edges deployed on unsupported hardware:
- Migrate the NSX appliances to a supported hardware generation. Refer to the Broadcom Compatibility Guide for supported hardware.
- If VMware EVC mode is disabled on the Cluster:
  - Best Practice is to perform the following operation on one appliance at a time.
    - Shut Down Guest OS for each NSX appliance one at a time (If the appliance does not shut down gracefully and gets hung, proceed to Power Off the appliance.)
    - Power On the NSX Appliance
      - NSX-T Managers: Power cycle one Manager node at a time to maintain cluster availability.
      - NSX-T Edges: Power cycle one Edge node at a time within an Edge cluster to prevent downtime. You may remediate multiple Edge nodes across different clusters simultaneously.
- If VMware EVC mode is enabled on the cluster hosting NSX appliances, follow the steps in 2.
NSX Manager and NSX Edges deployed on EVC-enabled clusters:
- Update the vSphere cluster EVC settings from the unsupported modes listed in environment section to a supported, newer CPU mode.
- On each vSphere Cluster where NSX Manager and NSX Edges are deployed, follow these steps:
  - Select the Cluster on the vCenter UI.
  - On "Configure" tab for the cluster, under "Configuration" section, select "VMware EVC".
  - Click on "EDIT" button
  - On the Change EVC Mode popup, change the CPU Mode according to the chipset of the hosts on the cluster:
    - For Intel processors, select CPU Mode Intel® "Broadwell" Generation or later.
    - For AMD processors, select CPU Mode AMD Zen Generation or later

- - Check if EVC mode is overridden at NSX Edge VM level, if it's in unsupported EVC mode then perform the following steps:
    - Update the NSX Edge VM EVC settings from the unsupported modes listed in environment section to a supported, newer CPU mode.
      - Select the NSX Edge VM on the vCenter Inventory tree.
      - On "Configure" tab for the VM, under "Settings" section, select "VMware EVC".
      - Click on "EDIT" button
      - On the Change EVC Mode popup, change the CPU Mode according to the chipset of the hosts on the cluster:
        For Intel processors, select CPU Mode Intel® "Broadwell" Generation or later.
        For AMD processors, select CPU Mode AMD Zen Generation or later.
  - Best Practice is to perform the following operation on one appliance at a time.
    - Shut Down Guest OS for each NSX appliance one at a time(If the appliance does not shut down gracefully and gets hung, proceed to Power Off the appliance.).
    - Power On the NSX Appliance
      - NSX-T Managers: Power cycle one Manager node at a time to maintain cluster availability.
      - NSX-T Edges: Power cycle one Edge VM at a time within an Edge cluster to prevent downtime. You may remediate multiple Edge nodes across different clusters simultaneously.

Retry and continue installation/upgrade after all NSX-T Managers and NSX Edges are remediated.

Additional Information

For more details on EVC levels, see Broadcom ERROR: The target host does not support the virtual machine’s current hardware requirements.