How to handle lost or stuck I/O on a host in vSAN cluster
search cancel

How to handle lost or stuck I/O on a host in vSAN cluster

book

Article ID: 326885

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Introduction:

  • Lost or wedged I/O is an I/O which is stuck outside of ESXi (device controller/firmware) that does not complete and doesn’t respond to abort and/or abort never completes.
  • Since the I/O is stuck outside of ESXi, the only option ESXi has is to send an abort. If the device/controller doesn’t respond to the abort within 120 seconds (default timeout) vSAN will take the disk/Disk Group to offline state to avoid affecting the entire vSAN cluster.

    Examples of Symptoms:



 

  • Check for Skyline Health Alarm "Operation Health" via SSH/Putty to any of the vSAN Hosts: 

    Run the following command:

    esxcli vsan health cluster get -t 'Operation health'

    Example of output: Operation health red

    Host      Disk      Overall health Metadata health Operational health In CMMDS/VSI OperationalState Description Recommendation UUID
    HOSTNAME  Disk(xxx) red            red             red                Yes     /Yes                  Stuck I/O is detected Migrate workload & power cycle host
    HOSTNAME  Disk(xxx) red            red             red                Yes     /Yes                  Stuck I/O is detected Migrate workload & power cycle host

  • In some scenarios, A Stuck IO can also results in VM crash event.

    Virtual machine vmware.log indicates the VM crash event

    "The guest has requested that the virtual machine be hard reset."

    vSAN reports heartbeat timeout events for VM namespaces.

    • /var/run/log/vobd.log : 

       [vmfsCorrelator] 2076401194729us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-############ ########-####-####-####-###########
       [vmfsCorrelator] 2076411926527us: [esx.problem.vmfs.heartbeat.recovered] ########-####-####-####-########### ########-####-####-####-###########

    • /var/run/log/hostd.log : 

       [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 17714 : Issue detected on ### in ##-#####: smartpqi01: pqisrc_wait_on_condition:0246: Controller is Offline 

  • The following are the stuck IO Logs events for reference:

    • If I/O is stuck or lost on the storage controller or the storage disk, the ESXi storage stack will try to abort them using the task management request displaying these console messages:

      YYYY-MM-DDThh:mm:ss.000Z cpu30:1001397101)ScsiDeviceIO: PsaScsiDeviceTimeoutHandlerFn:12834: TaskMgmt op to cancel IO succeeded for device naa.######## and the IO did not complete. WorldId 0, Cmd 0x28, CmdSN = 0x428.Cancelling of IO will be
      YYYY-MM-DDThh:mm:ss.000Z cpu30:1001397101)retried.

    • If such a lost I/O is found on a host, vSAN will offline the disk to ensure that it doesn't affect other hosts on the cluster as seen in

      • /var/run/log/vobd.log:

        YYYY-MM-DDThh:mm:ss.000Z: [vSANCorrelator] 19607827057us: [vob.vsan.lsom.stuckiooffline] vSAN device ########-########-####-####-####-############ detected stuck I/O error. Marking the device as offline.
        YYYY-MM-DDThh:mm:ss.000Z: [vSANCorrelator] 19607829404us: [esx.problem.vob.vsan.lsom.stuckiooffline] vSAN device ########-########-####-####-####-############ detected stuck I/O error. Marking the device as offline

      • /var/run/log/vmkernel.log:

        YYYY-MM-DDThh:mm:ss.000Z Wa(180) vmkwarning: cpu13:2101100)WARNING: MDR: MDRIOHOfflineDevice:4669: Offlining with IOLayer error 0xbad000a on t10.NVMe____#############___________________________################:#:##-###-#### handle
        YYYY-MM-DDThh:mm:ss.000Z In(182) vmkernel: cpu13:2101100)WOBTREE: IOLayerOfflineDevice:6570: IOLayerOfflineDevice t10.NVMe____#############___________________________################:#:# region ##-###-####-###-# status=I/O error
        YYYY-MM-DDThh:mm:ss.000Z In(182) vmkernel: cpu13:2101100)WOBTREE: IOLayer_SetDeviceOffline:8596: OfflineDevice t10.NVMe____#############___________________________################:# status=I/O error errType TRANSIENT
  • When Deduplication is not enabled: If the Cache Tier encounters stuck I/O the entire Disk Group it manages will be set to offline state.When Deduplication is enabled: If stuck I/O is detected on a disk, the entire Disk Group it manages will be set to offline state.

    YYYY-MM-DDThh:mm:ss.000Z: [vSANCorrelator] 19607827040us: [vob.vsan.lsom.stuckiopropagated] vSAN device ########-########-####-####-####-############ is under propagated stuck I/O error. Marking the device as offline.
    YYYY-MM-DDThh:mm:ss.000Z: [vSANCorrelator] 19607828405us: [esx.problem.vob.vsan.lsom.stuckiopropagated] vSAN device ########-########-####-####-####-############ is under propagated stuck I/O error. Marking the device as offline.
 
 

Environment

VMware vSAN 8.0.x

Resolution

  1. Migrate the workload and power cycle the host. After power cycle of the host, collect the vm-support along with driver/firmware logs.
     
  2. These issues are seen due to faulty hardware or firmware bugs.
     
  3. Proceed with opening a case with the hardware vendor.
     


Additional Information

Powered by Wolken Software