Linux Guest VM Filesystems Entering Read-Only State During Storage Array Upgrades

Products

• VMware vSphere ESXi

Issue/Introduction

• Linux Guest Operating System filesystems unexpectedly transition to a read-only state .
• The issue often correlates with backend storage maintenance or firmware upgrades.
• Virtual machine applications (e.g., databases, Splunk) fail due to an inability to write to disk.
• Log Insight does not show standard All Paths Down (APD) or Permanent Device Loss (PDL) events.
• In the /var/run/log/vmkernel.log of the affected ESXi hosts, entries show 0x4 0x44 0x0 sense data Failures on SAN

YYYY-MM-DDTHH:MM:SS.MSZ In(182) vmkernel: ScsiDeviceIO: 4697: Cmd(0x45db27bfb740) 0x88, CmdSN 0x800e000e from world 2110011 to dev "naa.################################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0

 

 

Environment

VMware vSphere ESXi 

Cause

This issue occurs when the storage array returns specific SCSI sense data indicating an internal hardware failure rather than a path failover Failures. When the ESXi host receives a hardware error status from the target, it propagates this to the Guest OS, which remounts the filesystem as read-only to prevent data corruption.

Interpretation of Sense Data:

• H:0x0 (Host Status = OK): The ESXi host storage adapter successfully transmitted the command.
• D:0x2 (Device Status = CHECK CONDITION): The storage array received the command but returned an error state.
• Sense Data 0x4 0x44 0x0: Indicates a Hardware Error and Internal Target Failure on the storage device Failures on SAN.

Resolution

Engage Storage Vendor: Provide the specific SCSI sense data (0x4 0x44 0x0) and timestamps to the storage vendor support team for investigation.