NSX Bare Metal Edge stops working after an extended uptime

Products

• VMware NSX

Issue/Introduction

1.  

   1. NSX T Bare metal edge stopped working and reboot is required to bring the edge into operational state
   2. NSX Bare Metal Edge has been working without issue for over 1000 days.
   3. NSX Bare Metal Edge is using AMD processors.
   4. NSX Bare Metal Edge stops functioning and console access to the server shows log lines such as:

    

   • watchdog: BUG: soft lockup - CPU### stuck for ##s! [process_name:##] audit backlog limit exceeded

Environment

VMware NSX
vCenter 
ESXi

Cause

Per AMD Erratum 1474, any physical server (regardless of the hardware manufacturer, such as Supermicro, Dell, HPE, etc.) may experience this issue under the following conditions:

1. 1. Processor is AMD EPYC 7532 32-Core Processor 
   2. Bare metal edge is powered ON  more than 1044 days

Resolution

Note: CPU power states and BIOS configurations are implemented, managed, and supported directly by the physical hardware vendor (e.g., HPE, Dell).

There are two workarounds referred by Dell in the referenced article

1. Disable 'cstate' in BIOS to prevent the CPU core entering cc6 state.
2. Reboot the system before it has an uptime of 1044 days. This could be a warm or cold reboot.

An AMD CPU core may stop responding after about 1044 days according to AMD erratum 1474