Unexpected shut down or crash of the Virtual Machine due to I/O Blockage and Resets.
search cancel

Unexpected shut down or crash of the Virtual Machine due to I/O Blockage and Resets.

book

Article ID: 432273

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Virtual machines protected by Dell EMC RecoverPoint may experience unexpected resets or Crash or Storage hangs. 
  • ESXI Host logs may contain similar errors as below

/var/run/log/vmkernel.log

YYYY-MM-DDTHH:MM:SS cpu26:2099998)esx_splitter: KL_INFO:862: #2 - ESXConfScanner_s_handleConfLineForListener:316: Debugging: esx.conf scan, /adv/Misc/HostIPAddr = ##.##.#.##
YYYY-MM-DDTHH:MM:SS  cpu26:2099998)esx_splitter: KL_INFO:862: #2 - ESXConfScanner_s_handleConfLineForListener:316: Debugging: esx.conf scan, /adv/Misc/HostName = ESXI_FQDN
YYYY-MM-DDTHH:MM:SS cpu1:5369897)esx_splitter: KL_INFO:862: #2 - EsxSplitterVolume_startIo: VAAI command 42 to protected volume guid 0x9bdee######bf4. reject with VMK_NOT_SUPPORTED
YYYY-MM-DDTHH:MM:SS cpu6:2099998)esx_splitter: KL_INFO:862: #2 - IoStats_s_printStats: total 6329 IOs over 60 seconds. average time to start 0us, pending 1230us, processing 8us
YYYY-MM-DDTHH:MM:SS cpu1:7475339)esx_splitter: KL_ERROR:937: #0 - IoEsx_ToStorage_s_forwardToLower: VSCSIFilter_IssueCommandToBackend Failed (io: 0x0), with status Would block
YYYY-MM-DDTHH:MM:SS cpu18:7508625)WARNING: iodm: IodmSasTransport:782: getSasPortStatistics failed for 'vmhba1' : Not implemented
YYYY-MM-DDTHH:MM:SS cpu39:5369897)esx_splitter: KL_INFO:862: #2 - EsxSplitterVolume_startIo: VAAI command 42 to protected volume guid 0x9bde#####7aabf4. reject with VMK_NOT_SUPPORTED

OR

[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu14:59359036)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu42:59359038)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu23:59359040)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu53:59359042)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-daemon-emcsplitter.so: filter open in cartel xxxxxxx (iofilterd) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu41:59359045)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-daemon-emcsplitter.so: filter open in cartel xxxxxxx (iofilterd) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu59:59359047)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-daemon-emcsplitter.so: filter open in cartel xxxxxxx (iofilterd) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu62:59359049)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu48:59359051)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu3:59359053)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel  xxxxxxx(vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu45:59359055)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu52:59359057)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall
[YYYY-MM-DDTHH:MM:SS] Al(177) vmkalert: cpu29:59359059)ALERT: /usr/lib64/vmware/plugin/libvmiof-disk-emcsplitter.so: filter open in cartel xxxxxxx (vmx) during uninstall

  • You may also see the Iofilter service getting crashed. 

[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu45:59359322)SchedVsi: 2208: Group: host/iofilters/emcsplitter(25625): max=-1 min=0 minLimit=-1, units: mb
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu45:59359322)SchedVsi: 2208: Group: host/iofilters/emcsplitter/daemon(25626): max=400 min=400 minLimit=400, units: mb
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu51:59242736)UserDump: 3157: iofltd-emcsplit: Dumping cartel xxxxxxx (from world 59242736) to file /var/core/iofilterd-zdump.xxx ...

  • VM log - /vmfs/volumes/Datastore/vmname/vmware.log may contain 

YYYY-MM-DDTHH:MM:SSZ vcpu-0 - CPU reset: hard (mode Emulation)
YYYY-MM-DDTHH:MM:SSZ vcpu-1 - CPU reset: hard (mode Emulation)

  • VM may also crash generating a VMX coredump. 

[YYYY-MM-DDTHH:MM:SS] Cr(01) worker-xxxxxxx - PANIC: Unexpected signal: 6.
[YYYY-MM-DDTHH:MM:SS] Wa(03) worker-xxxxxxx - A core file is available in "/vmfs/volumes/Datastore/vmname/vmx-zdump.xxx"

Environment

  • VMware vSphere ESXI 7.x
  • VMware vSphere ESXI 8.x
  • VMware vSphere ESXI 9.x

Cause

  •  The splitter acts as an interceptor between the VM and the storage. If the splitter encounters a synchronization issue or a communication failure with the RecoverPoint Appliance (RPA), it stops passing data to the storage backend to preserve data consistency. The splitter may reject storage acceleration commands (VAAI), forcing the guest OS/PVSCSI controller into a Busy or timeout state causing the VM to shut down.

  • The splitter may maintain stale locks on virtual disk files, causing "Conflict between buffered and unbuffered open" errors. This prevents the VM from accessing its own disks during power-on or migration.

Resolution

Powered by Wolken Software