Snapshot Consolidation of VM may fail with "Failed to get allocated sector chunks: Connection timed out"
search cancel

Snapshot Consolidation of VM may fail with "Failed to get allocated sector chunks: Connection timed out"

book

Article ID: 404071

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Snapshot Consolidation of Virtual Machine(VM) may fail with "Failed to get allocated sector chunks: Connection timed out"
  • Consolidation will be successful for the same VM upon migrating to another host
  • /vmfs/volumes/<datastore>/<VM_Folder>/vmware.log:

YYYY-MM-DDTHH:MM:SS In(05) vmx - SnapshotESXCombineProgressTotal: Snapshot consolidation progress: 79
YYYY-MM-DDTHH:MM:SS In(05) worker-2286529 - OBJLIB-FILEBE : FileBEIoctl: ioctl operation IOCTLCMD_VMFS_DELTADISKS(3033) failed on '/vmfs/devices/cbt/81a11e4-cbt' : Connection timed out (7208962)
YYYY-MM-DDTHH:MM:SS In(05) worker-2286529 - DISKLIB-VMFS  :VmfsSparseExtentCommonGetAllocatedSectorChunks: ObjLib_Ioctl failed 0x6e0009
YYYY-MM-DDTHH:MM:SS Er(02) worker-2286529 - DISKLIB-LINK  : DiskLinkGetAllocatedSectorChunks: Failed to get allocated sector chunks: Connection timed out
YYYY-MM-DDTHH:MM:SS Er(02) worker-2286529 - DISKLIB-LIB_MISC   : DiskLibGetAllocatedSectorChunksInRangeInt: failed to get allocated sector bitmap: Connection timed out
YYYY-MM-DDTHH:MM:SS Wa(03) worker-2286529 - Mirror: scsi0:0: Failed to get allocated sectors: Connection timed out.
YYYY-MM-DDTHH:MM:SS Wa(03) worker-2286529 - Mirror: scsi0:0: Mirror_SendBitmapToKernel: Failed to send allocated bitmap to VMkernel
YYYY-MM-DDTHH:MM:SS In(05) vmx - MirrorDiskCopyGetCopyProgress: Bitmap copy failed on disk /vmfs/volumes/<Datastore>/<VM_Folder>/<VM_Name>-000001.vmdk.
YYYY-MM-DDTHH:MM:SS In(05) vmx - ConsolidateDiskCopyCB: Mirror Disk copy failed on src disk: /vmfs/volumes/<Datastore>/<VM_Folder>/<VM_Name>-000001.vmdk and estination disk: /vmfs/volumes/<Datastore>/<VM_Folder>/<VM_Name>-000001.vmdk.
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Destroying virtual dev for scsi0:0 vscsi=9820528621658205
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - VMMon_VSCSIStopVports: No such target on adapter
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Closing disk 'scsi0:0'
YYYY-MM-DDTHH:MM:SS In(05) worker-2286529 - OBJLIB-FILEBE : FileBEIoctl: ioctl operation IOCTLCMD_VMFS_SVM(3044) failed on '/vmfs/devices/svm/81a11e4-89111e7-svmmirror' : Invalid argument (1441794)
YYYY-MM-DDTHH:MM:SS Wa(03) worker-2286529 - MirrorSendDiskBitmapIoctl: Cannot send source disk COW bitmap to kernel vmfs/volumes/<Datastore>/<VM_Folder>/<VM_Name>-000001.vmdk: Invalid argument
YYYY-MM-DDTHH:MM:SS Wa(03) worker-2286529 - Mirror: scsi0:0: Mirror_SendBitmapToKernel: Failed to send allocated bitmap to VMkernel

  • /var/log/vmkernel.log:

YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu20:2097818)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x2a (0x45b9429ab2c0, 2286520) to dev "<LUN ID>" on path "vmhba3:C0:T11:L0" Failed:
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu20:2097818)NMP: nmp_ThrottleLogForDevice:3898: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=############ CmdSN 0x161edc
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu20:2097818)ScsiDeviceIO: 4670: Cmd(0x45b9429ab2c0) 0x2a, cmdId.initiator=############ CmdSN 0x161edc from world 2286520 to dev "naa.################################" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu20:2097818)Cmd count Active:2 Queued:0
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu26:2153616)nfnic: <3>: INFO: fnic_taskMgmt: 2196: TaskMgmt abort sc->cdb: 0x28 sllid: 0xffffffffffffffff
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu26:2153616)nfnic: <3>: INFO: fnic_abort_cmd: 3874: Abort cmd called for Tag: 0x3c4  issued time: 40509 ms CMD_STATE: FNIC_IOREQ_CMD_PENDING CDB Opcode: 0x28  sc:0x45d97f4c6140 flags: 0x3 lun: 0 target: 0x4ba3c0
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu26:2153616)WARNING: nfnic: <3>: fnic_abort_cmd: 3889: Abort for cmd tag: 0x3c4 in pending state
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu25:2097459)nfnic: <3>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1870: io_req: 0x45d978018f08 sc: 0x45d97f4c6140 tag: 0x3c4 CMD_FLAGS: 0x53 CMD_STATE: FNIC_IOREQ_ABTS_PENDING ABTS pending hdr status: FCPIO_ABORTED scsi_status: 0x0$
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu25:2097459)nfnic: <3>: INFO: fnic_fcpio_itmf_cmpl_handler: 2468: io_req: 0x45d978018f08 sc: 0x45d97f4c6140 id: 0x3c4 CMD_FLAGS: 0x73 CMD_STATE: FNIC_IOREQ_ABTS_PENDINGhdr status: FCPIO_SUCCESS ABTS cmpl received
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu25:2097818)ScsiDeviceIO: 4670: Cmd(0x45d97f4c6140) 0x28, cmdId.initiator=############ CmdSN 0x161edb from world 2286520 to dev "naa.################################" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer.

Environment

VMware vSphere ESXi

Cause

The above issue is caused due to Fiber IO being aborted at the hardware layer resulting in failure of write operations. This issue can also be caused due to faulty Fiber SFP (Small Form-factor Pluggable) on the server.

Resolution

In order to resolve the issue, Proceed to engage the Hardware support team for SAN Switch and Storage to validate the faulty hardware.

Additional Information

In case if consolidation fails with "Device or resource busy" Refer to Consolidating snapshot failed: failed to get allocated sector bitmap with 'Device or resource busy'

Powered by Wolken Software