ESXi Host reports PSOD when "zloadmod.sh" script attempts to execute "cat /dev/ZDriverChr"

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi Host may encounter PSOD when "zloadmod.sh" script is executed on the host

YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]: #PF Exception 14 in world <>:cat IP <> addr <>

• /var/run/log/LogEFI.log (ESXi host)

YYYY-MM-DDTHH:HH:SS LogEFI: cpu36:<>)ESC[45mESC[33;1mVMware ESXi 8.0.3 [Releasebuild-24859861 x86_64]ESC[0m
YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]: #PF Exception 14 in world <>:cat IP <> addr <>
YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]: PTEs:0xc08006b023;0x8080036063;0x0;
YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]:
YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]: Module(s) involved in panic: [vmkernel Version Releasebuild-24859861]
YYYY-MM-DDTHH:HH:SS LogEFI: cpu36:<>)cr0=<> cr2=<> cr3=<> cr4=<>
YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]: *PCPU36:<>/cat
YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]: PCPU  0: SSSSUIISSSUSSISSUSSSSSSSUSSIUIUSSUISUUUSIUUUUISIVSSVVVVSVVUIVSVV
YYYY-MM-DDTHH:HH:SS LogEFI[<PID>]: PCPU 64: VUSVVSVVVVVSVVVVVVVVVVSVVVVVVVVV
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)Code start: <> VMK uptime: 0:00:xx:xx.xxx
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x453a0731ac50:[0x420017398a78]VMKAPICharDevDevfsWrapOpen@vmkernel#nover+0x48 stack: 0x38
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]CharDriverOpenDevice@vmkernel#nover+0x1c9 stack: 0x430edd814f40
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]FDS_OpenDeviceByDirOID@vmkernel#nover+0x195 stack: 0x453a0731adc8
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]DevFSFileOpen@vmkernel#nover+0x254 stack: 0x41ffd72bfce0
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]FSSVec_OpenFile@vmkernel#nover+0x18 stack: 0x0
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]FSSOpenFile@vmkernel#nover+0x1df stack: 0x453a0731b050
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]FSS_OpenFile@vmkernel#nover+0x2e stack: 0x0
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]FSS_LookupAndOpen@vmkernel#nover+0x120 stack: 0x4302c2e014a4
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]UserFileOpen@vmkernel#nover+0x718 stack: 0x453a0731b286
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]UserObjTraversePath@vmkernel#nover+0x159 stack: 0x9
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]UserObj_TraversePathAt@vmkernel#nover+0x8b stack: 0x100
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]UserObj_OpenAt@vmkernel#nover+0x54 stack: 0x0
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]UserObj_FDOpenAt@vmkernel#nover+0x88 stack: 0x433340802650
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]LinuxFileDesc_OpenAt@vmkernel#nover+0x11e stack: 0x30731bf40
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]User_LinuxSyscallHandler@vmkernel#nover+0xd9 stack: 0x3
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)0x<>:[0x<>]gate_entry@vmkernel#nover+0xa7 stack: 0x0
YYYY-MM-DDTHH:HH:SS LogEFI: cpu<id>:<>)base fs=0x0 gs=<> Kgs=0x0

• /var/run/log/vmkernel.log (on ESXi)
  
  grep "/tmp/zloadmod.sh" /var/run/log/vmkernel.log

YYYY-MM-DDTHH:HH:SS vmkernel: cpu##:<>)UserCartel: 2920: sh: Execute non-installed script: /tmp/zloadmod.sh
YYYY-MM-DDTHH:HH:SS vmkernel: cpu##:<>)UserCartel: 2920: sh: Execute non-installed script: /tmp/zloadmod.sh
YYYY-MM-DDTHH:HH:SS vmkernel: cpu##:<>)UserCartel: 2920: sh: Execute non-installed script: /tmp/zloadmod.sh
YYYY-MM-DDTHH:HH:SS vmkernel: cpu##:<>)UserCartel: 2920: sh: Execute non-installed script: /tmp/zloadmod.sh

Cause

PSOD on host was caused due to thread failing to access function when executing 'cat /dev/ZDriverChr'. This access was invoked by '/tmp/zloadmod.sh' which is part of Zerto Replication.

Resolution

In order to resolve the issue, proceed to engage Zerto Support as the issue is resolved with Zerto 10.0_U7P1. Refer to Release Notes for Zerto 10.0_U7P1 (under Resolved Issues - VMware)

Additional Information

In case if immediate upgrade of Zerto Solution is not feasible, proceed to reach out to Zerto support for workaround.