ESXi host crashes with a PSOD if seSparse snapshot created on base vmdk of size that is not 4K aligned
Article ID: 318661
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
- ESXi host crashes with a PSOD - @BlueScreen: PCPU 11: no heartbeat (3/3 IPIs received)
- You will see similar messages as below
@BlueScreen: PCPU 11: no heartbeat (3/3 IPIs received)
YYYY-MM-DDTHH:MM:SS.609Z cpu12:68535)@BlueScreen: PCPU 11: no heartbeat (3/3 IPIs received)
YYYY-MM-DDTHH:MM:SS.609Z cpu12:68535)Code start: 0x418037800000 VMK uptime: 115:12:44:05.065
YYYY-MM-DDTHH:MM:SS.609Z cpu12:68535)Saved backtrace from: pcpu 11 Heartbeat NMI
YYYY-MM-DDTHH:MM:SS.610Z cpu12:68535)0x43932041b770:[0x4180378698aa]DevFSIoctl@vmkernel#nover+0x1f6 stack: 0x43932041b7d0
YYYY-MM-DDTHH:MM:SS.610Z cpu12:68535)0x43932041b810:[0x41803783e344]FSSVec_Ioctl@vmkernel#nover+0x6c stack: 0x514c7428
YYYY-MM-DDTHH:MM:SS.610Z cpu12:68535)0x43932041b860:[0x41803783acaa]FSS_IoctlByFH@vmkernel#nover+0x9a stack: 0x19
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041b8a0:[0x418038a6bff9]CBT_Ioctl@(cbt)#<None>+0x201 stack: 0x0
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041b940:[0x418037869897]DevFSIoctl@vmkernel#nover+0x1e3 stack: 0x500735f55bb24bda
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041b9e0:[0x41803783e344]FSSVec_Ioctl@vmkernel#nover+0x6c stack: 0x416c742b
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041ba30:[0x41803783acaa]FSS_IoctlByFH@vmkernel#nover+0x9a stack: 0x43054028b940
YYYY-MM-DDTHH:MM:SS.612Z cpu12:68535)0x43932041ba70:[0x418037b66e09]VSCSIFsEmulateCommand@vmkernel#nover+0x15dd stack: 0x430013cb6bc0
YYYY-MM-DDTHH:MM:SS.612Z cpu12:68535)0x43932041bc70:[0x418037b680ec]VSCSI_FSCommand@vmkernel#nover+0xe4 stack: 0xffff8103c700dff0
YYYY-MM-DDTHH:MM:SS.612Z cpu12:68535)0x43932041bd40:[0x418037b5d5c5]VSCSI_IssueCommandBE@vmkernel#nover+0x31 stack: 0x43054028b940
YYYY-MM-DDTHH:MM:SS.613Z cpu12:68535)0x43932041bd80:[0x418037b5ea6c]VSCSIExecuteCommandInt@vmkernel#nover+0x3b4 stack: 0x823ac9000
YYYY-MM-DDTHH:MM:SS.613Z cpu12:68535)0x43932041be10:[0x418037b70d0b]LSIProcessReqInt@vmkernel#nover+0x6fb stack: 0x43932041beab
YYYY-MM-DDTHH:MM:SS.613Z cpu12:68535)0x43932041bf50:[0x418037b7142e]LSI_ProcessReq@vmkernel#nover+0x66 stack: 0x439334d27100
YYYY-MM-DDTHH:MM:SS.614Z cpu12:68535)0x43932041bf80:[0x4180379197d7]VMMVMKCall_Call@vmkernel#nover+0x157 stack: 0x43932041bfec
YYYY-MM-DDTHH:MM:SS.614Z cpu12:68535)0x43932041bfe0:[0x41803794b8a2]VMKVMM_ArchEnterVMKernel@vmkernel#nover+0xe stack: 0x0
YYYY-MM-DDTHH:MM:SS.618Z cpu12:68535)base fs=0x0 gs=0x418043000000 Kgs=0x0
YYYY-MM-DDTHH:MM:SS.609Z cpu12:68535)Saved backtrace from: pcpu 11 Heartbeat NMI
YYYY-MM-DDTHH:MM:SS.610Z cpu12:68535)0x43932041b770:[0x4180378698aa]DevFSIoctl@vmkernel#nover+0x1f6 stack: 0x43932041b7d0
YYYY-MM-DDTHH:MM:SS.610Z cpu12:68535)0x43932041b810:[0x41803783e344]FSSVec_Ioctl@vmkernel#nover+0x6c stack: 0x514c7428
YYYY-MM-DDTHH:MM:SS.610Z cpu12:68535)0x43932041b860:[0x41803783acaa]FSS_IoctlByFH@vmkernel#nover+0x9a stack: 0x19
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041b8a0:[0x418038a6bff9]CBT_Ioctl@(cbt)#<None>+0x201 stack: 0x0
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041b940:[0x418037869897]DevFSIoctl@vmkernel#nover+0x1e3 stack: 0x500735f55bb24bda
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041b9e0:[0x41803783e344]FSSVec_Ioctl@vmkernel#nover+0x6c stack: 0x416c742b
YYYY-MM-DDTHH:MM:SS.611Z cpu12:68535)0x43932041ba30:[0x41803783acaa]FSS_IoctlByFH@vmkernel#nover+0x9a stack: 0x43054028b940
YYYY-MM-DDTHH:MM:SS.612Z cpu12:68535)0x43932041ba70:[0x418037b66e09]VSCSIFsEmulateCommand@vmkernel#nover+0x15dd stack: 0x430013cb6bc0
YYYY-MM-DDTHH:MM:SS.612Z cpu12:68535)0x43932041bc70:[0x418037b680ec]VSCSI_FSCommand@vmkernel#nover+0xe4 stack: 0xffff8103c700dff0
YYYY-MM-DDTHH:MM:SS.612Z cpu12:68535)0x43932041bd40:[0x418037b5d5c5]VSCSI_IssueCommandBE@vmkernel#nover+0x31 stack: 0x43054028b940
YYYY-MM-DDTHH:MM:SS.613Z cpu12:68535)0x43932041bd80:[0x418037b5ea6c]VSCSIExecuteCommandInt@vmkernel#nover+0x3b4 stack: 0x823ac9000
YYYY-MM-DDTHH:MM:SS.613Z cpu12:68535)0x43932041be10:[0x418037b70d0b]LSIProcessReqInt@vmkernel#nover+0x6fb stack: 0x43932041beab
YYYY-MM-DDTHH:MM:SS.613Z cpu12:68535)0x43932041bf50:[0x418037b7142e]LSI_ProcessReq@vmkernel#nover+0x66 stack: 0x439334d27100
YYYY-MM-DDTHH:MM:SS.614Z cpu12:68535)0x43932041bf80:[0x4180379197d7]VMMVMKCall_Call@vmkernel#nover+0x157 stack: 0x43932041bfec
YYYY-MM-DDTHH:MM:SS.614Z cpu12:68535)0x43932041bfe0:[0x41803794b8a2]VMKVMM_ArchEnterVMKernel@vmkernel#nover+0xe stack: 0x0
YYYY-MM-DDTHH:MM:SS.618Z cpu12:68535)base fs=0x0 gs=0x418043000000 Kgs=0x0
- In the vmkernel.log you will see message as below related to seSparse snapshot created on base vmdk of size that is not 4K aligned - numBlocks=4080218930
YYYY-MM-DDTHH:MM:SS.582Z cpu18:2319716)VSCSI: 3801: handle 22158(vscsi0:1):Creating Virtual Device for world 2319506 (FSS handle 6974411622) numBlocks=4080218930 (bs=512)
YYYY-MM-DDTHH:MM:SS582Z cpu18:2319716)VSCSI: 273: handle 22158(vscsi0:1):Input values: res=0 limit=-2 bw=-1 Shares=1000
YYYY-MM-DDTHH:MM:SS cpu18:2319716)FDS: 586: Enabling IO coalescing on driver 'deltadisks' device '148851367-vmname-000001-sesparse.vmdk'
YYYY-MM-DDTHH:MM:SS.589Z cpu18:2319716)CBT: 1310: Created device 106c11369-cbt for cbt driver with filehandle 4408284009
YYYY-MM-DDTHH:MM:SS582Z cpu18:2319716)VSCSI: 273: handle 22158(vscsi0:1):Input values: res=0 limit=-2 bw=-1 Shares=1000
YYYY-MM-DDTHH:MM:SS cpu18:2319716)FDS: 586: Enabling IO coalescing on driver 'deltadisks' device '148851367-vmname-000001-sesparse.vmdk'
YYYY-MM-DDTHH:MM:SS.589Z cpu18:2319716)CBT: 1310: Created device 106c11369-cbt for cbt driver with filehandle 4408284009
Environment
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
Resolution
This issue is resolved in VMware ESXi 6.5 Update 3 and VMware ESXi 6.7 Update 3.
To work around this issue if you do not want to upgrade, specify the size of disk in GB or TB it would be 4K aligned (in other words, size is completely divisible by 4096)
In this example above here size of the base vmdk "vmname.vmdk" was 4080218930 sectors for 512 bytes each. If this size is rounded up to 4080218936 sectors it will be 4K aligned. By extending size of this vmdk to 4080218936 sectors this issue can be avoided.
To work around this issue if you do not want to upgrade, specify the size of disk in GB or TB it would be 4K aligned (in other words, size is completely divisible by 4096)
In this example above here size of the base vmdk "vmname.vmdk" was 4080218930 sectors for 512 bytes each. If this size is rounded up to 4080218936 sectors it will be 4K aligned. By extending size of this vmdk to 4080218936 sectors this issue can be avoided.
Feedback
Yes
No
Powered by
