ESXi host fails with PSOD "#PF Exception 14 in world 2101615:vvold-worker IP

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

Host in PSOD with the following backtrace:

Panic Message: @BlueScreen: #PF Exception 14 in world 2101615:vvold-worker IP 0x42001a908d81 addr 0x10

Backtrace:
 0x453ea651b538:[0x42001a908d81]SPLockWork@vmkernel#nover+0x15 stack: 0x41ffdd804600, 0x433904a0a660, 0x42001c82f2d6, 0x3c, 0x453ea651b5e8
 0x453ea651b540:[0x42001c82a10e]GetDevRefIfMatch@(vvol)#<None>+0x1f stack: 0x433904a0a660, 0x42001c82f2d6, 0x3c, 0x453ea651b5e8, 0x0
 0x453ea651b560:[0x42001c82f2d5]VVolDeviceIteratorNext@(vvol)#<None>+0x72 stack: 0x0, 0x1, 0x74, 0x42001c8347ad, 0xffffffffffffffff
 0x453ea651b590:[0x42001c8347ac]VVolVsi_GetUnbindVVols@(vvol)#<None>+0x6d stack: 0xffffffffffffffa0, 0xf00000000, 0x74, 0x0, 0x74
 0x453ea651b5e0:[0x42001a8027b4]VSI_GetListInfo@vmkernel#nover+0x265 stack: 0x41ffdd800074, 0x42001c834740, 0x42001a8027a7, 0x0, 0x4cf52069a0
 0x453ea651b650:[0x42001ad2974a]UW64VMKSyscallUnpackVSI_GetList@vmkernel#nover+0x1b7 stack: 0x4d3db3b424, 0x0, 0x138b, 0x74, 0x0
 0x453ea651bee0:[0x42001acb36ea]User_UWVMK64SyscallHandler@vmkernel#nover+0x183 stack: 0x0, 0x4cf4ebcd60, 0x42001a954b64, 0x0, 0x4d3db3b3f0
 0x453ea651bf40:[0x42001a94b258]SyscallUWVMK64@vmkernel#nover+0x90 stack: 0x0, 0x0, 0x4d366a0fb8, 0x0, 0x0

 

Environment

VMware vSphere ESXi 7.0

Cause

Right before the fiter->device got corrupted and the ESXi host hit PSOD, we see that vVol naa.xxx was removed from the device table. 

2023-09-15T06:32:46.677Z cpu64:11102441 opID=5a885b75)osfs: OSFS_Lookup:2779: Lookup error: file = naa.xxx, status = Failure
2023-09-15T06:32:46.677Z cpu144:21222939)VVol: VVolAllocAndInitDevice:8187: Creating a device for naa.xxx (Type VVOL_OBJTYPE_CONFIG)
2023-09-15T06:32:46.835Z cpu144:21222939)VVol: VVolRemoveDev:7335: Unlinking (VVOL_OBJTYPE_CONFIG) VVol device naa.xxx
2023-09-15T06:32:46.835Z cpu146:2101615)World: 3072: PRDA 0x420064800000 ss 0x0 ds 0x10b es 0x10b fs 0x10b gs 0x0
2023-09-15T06:32:46.835Z cpu146:2101615)World: 3074: TR 0xf58 GDT 0x453e00124000 (0xf77) IDT 0x42001a950000 (0xfff)
2023-09-15T06:32:46.835Z cpu146:2101615)World: 3075: CR0 0x80010031 CR3 0x302170b3000 CR4 0x142768
2023-09-15T06:32:46.867Z cpu146:2101615)Backtrace for current CPU #146, worldID=2101615, fp=0x0
2023-09-15T06:32:46.867Z cpu146:2101615)0x453ea651b538:[0x42001a908d81]SPLockWork@vmkernel#nover+0x15 stack: 0x41ffdd804600, 0x433904a0a660, 0x42001c82f2d6, 0x3c, 0x453ea651b5e8

If a device does not meet the filter criteria, it is not referenced, and the hash bucket lock is subsequently released. Later, when attempt to retrieve the next device, encounter an issue because the device has already been removed and its memory freed. This results in accessing memory that has been freed.

Resolution

 

• This fix is available in version 8.0U1/U2

• The fix will also be backported to the 7.P09 patch branch.