Guest OS IO Error /TIME OUT because of backend disk issue
search cancel

Guest OS IO Error /TIME OUT because of backend disk issue

book

Article ID: 415088

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Guest OS can not read /write disk on VMFS, reporting I/O error and DID_TIME_OUT
    kernel: blk_update_request: I/O error, dev sde, sector 22487864 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
kernel: sd 0:0:0:0: [sdn] tag#570 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=461s
  • vmware.log reported io aborting at the same time.
    In(05) vcpu-10 - PVSCSI: scsi0:4: aborting cmd 0x289
In(05) vcpu-10 - PVSCSI: scsi0:4: aborting cmd 0x2fc
  • vmkernel.log reported io aborting at the same time.
    
cpu18:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x289
cpu2:2097236)ScsiDeviceIO: 4124: Cmd(0x45b96633bf88) 0x2a, CmdSN 0x289 from world 2144938 to dev "naa.xxx" failed H:0x5 D:0x0 P:0x0
cpu18:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x2fc
cpu2:2097236)ScsiDeviceIO: 4124: Cmd(0x45d9544b3b88) 0x2a, CmdSN 0x2fc from world 2144938 to dev "naa.xxx" failed H:0x5 D:0x0 P:0x0
cpu18:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x3e9
 cpu2:2097236)ScsiDeviceIO: 4124: Cmd(0x45d954508188) 0x8a, CmdSN 0x3e9 from world 2144938 to dev "naa.xxx" failed H:0x5 D:0x0 P:0x0
cpu18:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x1c1
cpu2:2097236)ScsiDeviceIO: 4124: Cmd(0x45b96643da88) 0x2a, CmdSN 0x1c1 from world 2144938 to dev "naa.xxx" failed H:0x5 D:0x0 P:0x0
cpu18:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x1d9
cpu2:2097236)ScsiDeviceIO: 4124: Cmd(0x45d9544b8188) 0x88, CmdSN 0x1d9 from world 2144938 to dev "naa.xxx" failed H:0x5 D:0x0 P:0x0
cpu18:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x3bd
cpu18:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x22b
cpu1:2144951)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x2a6
cpu21:2144953)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x39d
cpu21:2144953)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x274
cpu21:2144953)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x24a
 cpu21:2144953)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x3c8
cpu21:2144953)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x348
cpu2:2097239)ScsiDeviceIO: 4124: Cmd(0x45b96937fcc8) 0x8a, CmdSN 0x348 from world 2144938 to dev "naa.xxx" failed H:0x5 D:0x0 P:0x0
cpu21:2144953)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x39b
cpu2:2097239)ScsiDeviceIO: 4124: Cmd(0x45b9665f9688) 0x8a, CmdSN 0x39b from world 2144938 to dev "naa.xxx" failed H:0x5 D:0x0 P:0x0
cpu21:2144953)PVSCSI: 2698: scsi0:4: SCSI ABORT ctx=0x28e
  • naa.xxx 's performande is very bad.
    cpu27:2098083)ScsiDeviceIO: 1513: Device naa.xxxperformance has improved. I/O latency reduced from 11478657 microseconds to 2221645 microseconds.
cpu16:2098078)WARNING: ScsiDeviceIO: 1513: Device naa.xxxperformance has deteriorated. I/O latency increased from average value of 3645 microseconds to 18369821 microseconds.

Environment

ESXi 

Cause

Bad disk under RAID5

Resolution

  1. Check with hardware vendor to fix the disk issue
  2. Fix Guest OS issue with OS vendor or restore from backup
Powered by Wolken Software