ESXi 7.0 U2a host experiences PSOD during storage rescan with references "SCSI path sc IP / SCSI path scan helpers" in backtrace.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi host fails with PSOD during storage rescan task with high number of CPU counts.
• Before PSOD backtrace, we see storage heap messages.

WARNING: Heap: 3707: Heap storageHeap (347009448/347079080): Maximum allowed growth (69632) too small for size (94208)ESC[0m WARNING: Heap: 3707: Heap storageHeap (347009448/347079080): Maximum allowed growth (69632) too small for size (147456)ESC[0m WARNING: Heap: 4078: Heap_Align(storageHeap, 143520/143520 bytes, 8 align) failed.  caller: 0x42000041de0dESC[0m

• You see a backtrace similar to:​​​​

cpu108:2098227)World: 3227: PRDA 0x42005b000000 ss 0x0 ds 0xf50 es 0xf50 fs 0xf50 gs 0xf50
cpu108:2098227)World: 3229: TR 0xf58 GDT 0x4539000d8000 (0xf77) IDT 0x42000014b000 (0xfff)
cpu108:2098227)World: 3230: CR0 0x8001003d CR3 0x6c638000 CR4 0x10216c
cpu108:2098227)Backtrace for current CPU #108, worldID=2098227, fp=0x0
cpu108:2098227)0x4539a199b910:[0x42000041eba5]SCSIStartPathCommands@vmkernel#nover+0x6f1 stack: 0x4309920ddd40, 0x4309a63edda0, 0x0, 0x147a0e7c588, 0x42005b005680
cpu108:2098227)0x4539a199ba40:[0x4200004249af]SCSIIssueAsyncPathCommandDirect@vmkernel#nover+0x2c8 stack: 0x4539a199baf8, 0x28, 0x4539a199bac0, 0x4539a199ba70, 0x35
cpu108:2098227)0x4539a199bb50:[0x420000425833]SCSIIssueAsyncMPPPathCommand@vmkernel#nover+0xe0 stack: 0x3, 0x420000103caa, 0x4539a199bbb8, 0x4539a199bbb8, 0x4309a63ed5c0
cpu108:2098227)0x4539a199bba0:[0x42000042592c]SCSIIssueSyncPathCommand@vmkernel#nover+0xa9 stack: 0x6c415f7061654820, 0x0, 0x1000000, 0x7900000100, 0x6220303235333431
cpu108:2098227)0x4539a199bc10:[0x420000425afd]SCSISyncPathCmdWithRetriesInt@vmkernel#nover+0x5e stack: 0xff00000000, 0x147a0e7c108, 0x0, 0x192000ea0, 0x430992000a78
cpu108:2098227)0x4539a199bc80:[0x420000425c35]SCSISyncPathCmdWithRetries@vmkernel#nover+0xe stack: 0x0, 0x420000428d23, 0x4309a63ed5c0, 0x0, 0x4539a199bd61
cpu108:2098227)0x4539a199bca0:[0x420000428d22]SCSIDoStandardInquiry@vmkernel#nover+0x57 stack: 0x4539a199bd61, 0x420000429675, 0x4539a199bd18, 0x4539a199bcd0, 0x17000
cpu108:2098227)0x4539a199bcc0:[0x420000429674]SCSIScanPath@vmkernel#nover+0x25 stack: 0x17000, 0xfee, 0x430992000010, 0x42000013bb9d, 0x0
cpu108:2098227)0x4539a199bea0:[0x420000429e72]SCSIScanLUNsHelper@vmkernel#nover+0x393 stack: 0x4309a64b35a0, 0x420000104296, 0x113b00000000, 0x4309a63ed5c0, 0x4309a64b002c
cpu108:2098227)0x4539a199bf30:[0x4200000d76c0]HelperQueueFunc@vmkernel#nover+0x285 stack: 0x43099208f958, 0x43099208f980, 0x4539a19a1000, 0x43099208f918, 0x4539a181bf60
cpu108:2098227)0x4539a199bfe0:[0x420000381ead]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x4200000c2c24, 0x0, 0x0, 0x0
cpu108:2098227)0x4539a199c000:[0x4200000c2c23]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
cpu108:2098227)ESC[45mESC[33;1mVMware ESXi 7.0.2 [Releasebuild-17867351 x86_64]ESC[0m
 2098227:SCSI path sc IP 0x42000041eba5 addr 0x13690

Environment

VMware vSphere ESXi 7.0.x

Cause

ESXi host is running out of storage heap.

Resolution

Issue is resolved in vSphere 7.0 P03 (Update 2c).