ESXi Host reports PSOD due to issues with ZDriverChr (Zerto)

Products

• VMware vSphere ESXi

Issue/Introduction

PSOD encountered when ESXi uses zdriver by Zerto

Panic Message: @BlueScreen: #PF Exception 14 in world 2143148:cat IP 0x4200105983c8 addr 0x41ffe5c0be60

 Backtrace: 

#0  0x00004200105983c8 in VMKAPICharDevOpen (handle=0x43101ba0f470, handle=0x43101ba0f470, flags=1) at bora/vmkernel/main/vmkapi_char.c:461
#1  VMKAPICharDevDevfsWrapOpen (handle=0x43101ba0f470, flags=1) at bora/vmkernel/main/vmkapi_char.c:1383
#2  0x0000420010458ab5 in CharDriverOpenDevice (worldID=<optimized out>, dirOID=<optimized out>, deviceName=<optimized out>, flags=1, deviceHandleID=0x430e53a4fe00) at bora/vmkernel/filesystems/devices/charDriver.c:872
#3  0x0000420010abf175 in FDS_OpenDeviceByDirOID (worldID=2143148, driverID=0x41ffd04bf9a0, dirOID=dirOID@entry=0x453b2449ae60, deviceName=deviceName@entry=0x453b2449ade0 "ZDriverChr", flags=1, pfdsHandle=pfdsHandle@entry=0x453b2449add8) at bora/vmkernel/filesystems/devices/fsDeviceSwitch.c:608
#4  0x000042001045439c in DevFSFileOpen (fileData=0x430e59fbcf80, fhID=14563916, openFlags=1, dataIn=<optimized out>) at bora/vmkernel/filesystems/devfs/devfs.c:4067
#5  0x0000420010441f90 in FSSVec_OpenFile (desc=<optimized out>, fhID=<optimized out>, openFlags=<optimized out>, dataIn=<optimized out>) at bora/vmkernel/filesystems/fsSwitchVec.c:444
#6  0x0000420010ab6f13 in FSSOpenFile (fileDesc=0x430e59fbcf80, openFlags=1, fileHandleID=0x453b2449b270, dataIn=0x0) at bora/vmkernel/filesystems/fsSwitch.c:7343
#7  0x0000420010aba87a in FSS_OpenFile (fileOID=<optimized out>, openFlags=1, fileHandleID=0x453b2449b270) at bora/vmkernel/filesystems/fsSwitch.c:3939
#8  0x0000420010abaa60 in FSS_LookupAndOpen (parentOID=parentOID@entry=0x430e59fa3630, childName=childName@entry=0x453b2449bd40 "ZDriverChr", openFlags=openFlags@entry=1, fileHandleID=fileHandleID@entry=0x453b2449b270) at bora/vmkernel/filesystems/fsSwitch.c:4063
#9  0x00004200109e562c in UserFileOpen (parent=0x45db170c7b40, arc=0x453b2449bd40 "ZDriverChr", objFlags=0, mode=<optimized out>, objOut=0x453b2449b400) at bora/vmkernel/user/userFile.c:668
#10 0x00004200109ff1b9 in UserObjTraversePath (uci=uci@entry=0x43343b802010, obj=0x45db170c7b40, path=path@entry=0x43343b80ab20 "/dev/ZDriverChr", flags=flags@entry=0, vmkacPerm=vmkacPerm@entry=1, mode=mode@entry=0, objOut=0x453b2449bd38, arc=<optimized out>, maxArcLen=256, oldObj=0x0, symLinkLimit=<optimized out>) at bora/vmkernel/user/userObj.c:4094
#11 0x00004200109ff6c7 in UserObj_TraversePathAt (uci=uci@entry=0x43343b802010, path=path@entry=0x43343b80ab20 "/dev/ZDriverChr", flags=flags@entry=0, vmkacPerm=1, vmkacPerm@entry=0, mode=mode@entry=0, objOut=objOut@entry=0x453b2449bd38, arc=0x453b2449bd40 "ZDriverChr", arcLen=256, oldParent=0x0, startAt=0x45bb1a06c5a0) at bora/vmkernel/user/userObj.c:3790
#12 0x0000420010a004e0 in UserObj_TraversePathAt (startAt=0x45bb1a06c5a0, oldParent=0x0, arcLen=256, arc=0x453b2449bd40 "ZDriverChr", objOut=0x453b2449bd38, mode=0, vmkacPerm=0, flags=0, path=0x43343b80ab20 "/dev/ZDriverChr", uci=0x43343b802010) at bora/vmkernel/user/userObj.c:4232
#13 UserObj_OpenAt (uci=uci@entry=0x43343b802010, path=path@entry=0x43343b80ab20 "/dev/ZDriverChr", flags=flags@entry=0, vmkacPerm=vmkacPerm@entry=1, mode=mode@entry=0, objOut=objOut@entry=0x453b2449be78, startAt=0x45bb1a06c5a0) at bora/vmkernel/user/userObj.c:4232
#14 0x0000420010a00584 in UserObj_FDOpenAt (uci=uci@entry=0x43343b802010, fd=fd@entry=0x453b2449bed4, vmkPath=0x43343b80ab20 "/dev/ZDriverChr", flags=0, vmkacPerm=1, mode=mode@entry=0, startAt=0x45bb1a06c5a0) at bora/vmkernel/user/userObj.c:3076
#15 0x00004200109c4ede in LinuxFileDesc_OpenAt (dirfd=-100, userPath=<optimized out>, flags=<optimized out>, mode=0) at bora/vmkernel/user/linuxFileDesc.c:979
#16 0x00004200109cfddd in User_LinuxSyscallHandler (fullFrame=0x453b2449bf40) at bora/vmkernel/user/user.c:2109
#17 0x0000420010a9b0c7 in gate_entry () at bora/vmkernel/main/x86/gates64.S:175
#18 0x0000002cde3a0c58 in ?? ()

 

Cause

 PSOD occurred because the callback routine for the character device /dev/ZDriverChr was not allocated or initialized. 

Resolution

The Zerto team should be engaged to analyze the memory dump for any potential abnormalities or issues related to their appliances or services running on the ESXi host.

 

Additional Information

This is commonly observed when multiple Zerto appliances are running on the same ESXi host, which can lead to resource contention or conflicts which can lead to a PSOD situation.