PSOD "#GP Exception 13 in world ###:vmotionRecvH IP ###" during vMotion on ESX 9.1

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• While migrating a virtual machine (VM) from or to an ESX 9.1 host using vMotion or Cross-vCenter vMotion, the host might randomly fail with a Purple Screen (PSOD) showing the following exception:

  #GP Exception 13 in world ###:vmotionRecvH IP ###

  Similar to the following example:
  
• The host has multiple of its physical network cards configured as uplinks for the vMotion network
• These physical network adapters are 25 Gigabit or 100 Gigabit cards

Environment

VMware ESX 9.1

Cause

In order to handle vMotion traffic, ESX creates a number of "stream helpers", maximum 7 per physical uplink. The specific number of these depending on the overall network capacity in the vMotion network, but mainly on the number of the physical uplinks and their bandwidth.
For example, in a configuration with 4 * 100GBit Network cards (NICs) used as uplinks for the vMotion vmkernel adapters the number of these stream helpers might reach 28 overall.
However, a defect was found in the vMotion statistics tracking logic in ESX 9.1, which can cause a buffer overflow if the number of stream helpers exceeds 16.

A defect in the vMotion statistics tracking logic causes a buffer overflow when the number of "stream helpers" exceeds 16. In environments with 4x 100GbE NICs, the system scales to 28 helpers, triggering the crash.

Resolution

A permanent fix is not available at present, the VMware engineering team is working to implement a fix in a future release.

To Workaround the issue apply one of the below options:

1. Configure a cap/limit for the number of vMotion stream helpers (recommended)

   Set a fixed maximum number of stream helpers using an avanced host option. This is the preferred method as it does not require any changes to the actual physical network configuration:
   1. Open an SSH connection with the ESX host and log in using the root account.
   2. Run the following command to limit the number of vMotion stream helpers to 16:

      # esxcli system settings advanced set -o /Migrate/VMotionStreamHelpers -i 16

   3. Confirm the current setting:

      # esxcli system settings advanced list -o /Migrate/VMotionStreamHelpers

   Note: Alternatively vSphere Client can be used to access and reconfigure the ESX advanced options. See Configuring advanced options for ESXi for more information.

2. Reduce the uplink count on the vMotion network

   As mentioned further above, ESX will create 7 stream helpers per physical NICs at maximum. In order to reliably prevent the PSOD, there must not be more than 16 stream helpers at a time.
   By reducing the number of physical uplinks used by the on the vMotion network(s) down to 2, the number of stream helpers will never grow beyond 14, thus keeping the number below the threshold for the PSOD.