vSAN host enters PSOD state sporadically when Storage Accelerator and virtual NVMe controllers are used

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• You are running ESXi 7.0 U3c or newer.
• You are seeing different hosts sporadically entering PSOD within the same vSAN cluster.
• You have "Storage Accelerator" (CBRC) of Horizon enabled in the cluster.
• One or more virtual machines using the virtual NVMe Controller are located on the vSAN datastore.
• The virtual machine using the virtual NVMe Controller has at least one snapshot.
• The PSOD stacktrace shown is very similar to:

[e-mail]#0.0.0.1+0x12dc stack: 0x45d9800d1080
[e-mail]#0.0.0.1+0x990 stack: 0x431e3e6c4070
[e-mail]#0.0.0.1+0x16 stack: 0x0
[e-mail]#0.0.0.1+0x9c4 stack: 0xc000000
[e-mail]#0.0.0.1+0x19 stack: 0x453967a9bf10
[e-mail]#0.0.0.1+0x4a stack: 0x45d9c61f7448
[e-mail]#0.0.0.1+0x637 stack: 0x1e6580a69fe0f2
vmkWorldFunc@vmkernel#nover+0x49 stack: 0x42000751dd6c
CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0

Environment

VMware vSAN 7.0.x

Cause

When above symptoms match, during NVMe SCSI emulation a unexpected state caused the host not to be able to correctly process the I/O.

Resolution

This has been addressed in ESXi 7.0 U3i.

Workaround:
Use a different virtual storage controller on your virtual machines other than virtual NVMe controller.