ESXi Host hits PSOD during vMotion in NSX

Products

• VMware NSX

Issue/Introduction

• In VMware NSX environments, an ESXi host may unexpectedly fail or trigger a Purple Screen of Death (PSOD) during a virtual machine vMotion.

• This issue is typically observed in scaled environments when performing vMotion on VMs that have switch security applied to their ports. The failure happens during the VM or vNIC port disconnect and reconnect phases of the vMotion process.

• When analyzing the crash logs or the host screen, the following backtrace is observed:
    0x453b0cd9bbe0:[0x42002c7dd49d][email protected]#1.0.9.1.25524171+0x3d6 stack: 0x60006c3, 0x453b0cd9bc60, 0x60006c3, 0x0, 0x0
  0x453b0cd9bc80:[0x42002c5f4522][email protected]#1.0.9.1.25524171+0x6f stack: 0x43171ac00000, 0x4307ba3ee218, 0x4307ba3ee218, 0x4328ebf22c10, 0x60006c3

Environment

VMware NSX

Cause

This issue is caused by a race condition that occurs during the port disconnect operation.

Specifically, a timing conflict arises when a timer callback is actively running in the background at the exact same time the port disconnect callback is attempting to stop and clean up that timer. Because this is a highly specific timing issue tied directly to how switch security filters are destroyed during the vMotion disconnect phase, it triggers a CPU scheduler failure.

Resolution

This is a known issue. The race condition has been permanently resolved in the following releases:

• NSX 4.2.5

• NSX 9.1.1

Upgrade to one of these versions (or later) to prevent the issue from occurring.

Workaround and Recovery:

Because this is a strict race condition driven by background timing during standard vMotion port disconnect operations, there is no preventative workaround to avoid the issue on affected versions.

If a host runs into this assertion, you must perform the following graceful recovery steps:

1. Reboot the affected ESXi host.

2. Once the host has successfully booted and rejoined the cluster, manually power on the affected virtual machines.