VMs in read only state from Guest OS
search cancel

VMs in read only state from Guest OS

book

Article ID: 393549

calendar_today

Updated On:

Products

Issue/Introduction

Symptom:

  • VMs go into read only state or freezes from Guest OS layer.

  • VMs are unable to write to storage from Guest OS.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

  • Frame drops and device busy errors are seen due to excessive unmap request from the storage which leads to host to storage connectivity issue, affecting the VMs running on the host.

  • SSH to the host where the VMs are registered and navigate to "var/run/log/vmkernel.log" to find below events -

    2025-03-20T11:44:04.075Z cpu36:2098423)NMP: nmp_ThrottleLogForDevice:3875: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430915bede80 CmdSN 0x420a35
    2025-03-20T11:44:04.075Z cpu36:2098423)ScsiDeviceIO: 4124: Cmd(0x45b982394708) 0x28, CmdSN 0x420a35 from world 4420154 to dev "naa.################################" failed H:0x2 D:0x0 P:0x0
    2025-03-20T12:32:53.767Z cpu31:2098421)ScsiDeviceIO: 4124: Cmd(0x45b984e9e7c8) 0x42, CmdSN 0xf0fc12 from world 4425760 to dev "naa.################################" failed H:0x0 D:0x8 P:0x0
    2025-03-20T12:32:53.768Z cpu10:2098423)lpfc: lpfc_handle_status:5637: 1:(0):3271: FCP cmd x42 failed <3/1> sid x20a2c0, did x20b940, oxid x1cd0 iotag x7f6 SCSI Busy -
    2025-03-24T22:45:43.846Z cpu6:2098421)ScsiDeviceIO: 4124: Cmd(0x45b98505e708) 0x2a, CmdSN 0x42 from world 2099966 to dev "naa.################################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0
    2025-03-25T05:31:01.721Z cpu41:2098423)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x42 (0x45b9802a7948, 4425760) to dev "naa.################################" on path "vmhba5:C0:T3:L1" Failed:

    To understand the above highlighted sense codes -

    Type Code Description
    Host Status 0x2 This status is returned when the HBA driver is unable to issue a command to the device. This status can occur due to dropped FCP frames in the environment.
    OP Code 0x28 Host is unable to read the lun.
    OP Code 0x2a Host is unable to write on the lun.
    OP Code 0x42 Unmap request from the storage.


    Note: Refer below KBs to find the above SCSI sense code documentation.

    Interpreting SCSI sense codes in VMware ESXi

    SCSI Common Codes & Values

Resolution

  • Reach out to the storage vendor to check if we should disable unmap on the datastores by following - How to throttle the unmap requests on Datastore ( Space Reclamation )

    Steps - vCenter vSphere UI > Datastore View > Select the Datastore > Configure > General > Space  Reclamation  >Edit 


    Click Edit > Select the Reclamation rate from drop down.




    Note -
    • VMware does not recommend to disable or modify the "unmap" operations unless otherwise specified by the storage vendor.

    • The changes in default "Space Reclamation Settings" may deviate from the system defaults and can have other unintended behaviors.

Additional Information

Powered by Wolken Software