Linux VMs flags their file-system in read-only after datastore inaccessibility

Products

• VMware vSphere ESXi

Issue/Introduction

• Network switch outage caused iSCSI/NFS storage outage.
  
  
• Backend storage outage [Fabric connectivity issues, Storage volume issue] 
  
  
• All hosts report uplink down issue in /var/run/log/vobd.log:
  YYYY-MM-DDTHH:MM:SS: [netCorrelator] ################: [vob.net.dvport.uplink.transition.down] Uplink: vmnic1 is down. Affected dvPort: #2/5# 00 4f d2 a4 05 ## ## ## ## ## ## ## ## ## ##. 1 uplinks up. Failed criteria: 128
YYYY-MM-DDTHH:MM:SS: [netCorrelator] ################: [vob.net.dvport.uplink.transition.down] Uplink: vmnic4 is down. Affected dvPort: #7/5# 00 4f d2 a4 05 ## ## ## ## ## ## ## ## ## ##. 1 uplinks up. Failed criteria: 128 
  
  
• File-system on few Linux VMs is set to read-only state post-restoration of the network issue.
  
  
• Namespace objects of the impacted virtual machines report time-out errors  in /var/run/log/vobd.log: 
  YYYY-MM-DDTHH:MM:SS: [vmfsCorrelator] 26764193007100us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-#########6ac 54#####-########-####-#########6ac
YYYY-MM-DDTHH:MM:SS: [vmfsCorrelator] 26764193007176us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-#########6ac 67#####-########-####-#########6ac
YYYY-MM-DDTHH:MM:SS: [vmfsCorrelator] 26764193007212us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-#########6ac 31#####-########-####-#########6ac
  
  
• Following is the vmware.log of an affected VM showing vmware tools timeout and error writing log:
  YYYY-MM-DDTHH:MM:SS No(00) vmx - >>> Error writing log, 178 bytes discarded. Disk full?
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: Tools heartbeat timeout.
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: Running status rpc handler: 1 => 0.
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: Changing running status: 1 => 0.
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value 17560572 (last received 21s ago)
YYYY-MM-DDTHH:MM:SS No(00) vcpu-7 - >>> Error writing log, 87 bytes discarded. Disk full?
YYYY-MM-DDTHH:MM:SS No(00) vmx - >>> Error writing log, 106 bytes discarded. Disk full?
YYYY-MM-DDTHH:MM:SS In(05) vmx - GuestRpc: GuestRpcResetVsockChannel: channel 1
YYYY-MM-DDTHH:MM:SS In(05) vmx - GuestRpc: Closing channel 1 connection 3 

Environment

vSphere 7.x

vSphere 8.x 

Cause

• Due to unavailability of the backend storage, the affected Linux VMs change the state of its filesystem to read- to prevent any sort of filesystem corruption.

Resolution

• After recovery of the storage connectivity and restoring of datastore/s to functional state, reboot of the Linux VMs to restore original state of filesystem from read-only.
  
  
• If reboot does not fix, mount the filesystem manually using command:
  # mount -o remount /
  
  
  

Additional Information

• Refer following document for details on why Linux VMs file system go read-only :
  https://knowledge.broadcom.com/external/article/343025/linux-based-file-systems-become-readonly.html