Linux VMs flags their file-system in read-only after datastore inaccessibility
search cancel

Linux VMs flags their file-system in read-only after datastore inaccessibility

book

Article ID: 408670

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware Cloud Director

Issue/Introduction

  • Network switch outage caused iSCSI/NFS storage outage.

  • Backend storage outage [Fabric connectivity issues, Storage volume issue] 

  • All hosts report uplink down issue in /var/run/log/vobd.log:
    YYYY-MM-DDTHH:MM:SS: [netCorrelator] ################: [vob.net.dvport.uplink.transition.down] Uplink: vmnic1 is down. Affected dvPort: #2/5# 00 4f d2 a4 05 ## ## ## ## ## ## ## ## ## ##. 1 uplinks up. Failed criteria: 128
    YYYY-MM-DDTHH:MM:SS: [netCorrelator] ################: [vob.net.dvport.uplink.transition.down] Uplink: vmnic4 is down. Affected dvPort: #7/5# 00 4f d2 a4 05 ## ## ## ## ## ## ## ## ## ##. 1 uplinks up. Failed criteria: 128 

  • File-system on few Linux VMs is set to read-only state post-restoration of the network issue.

  • Namespace objects of the impacted virtual machines report time-out errors  in /var/run/log/vobd.log: 
    YYYY-MM-DDTHH:MM:SS: [vmfsCorrelator] 26764193007100us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-#########6ac 54#####-########-####-#########6ac
    YYYY-MM-DDTHH:MM:SS: [vmfsCorrelator] 26764193007176us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-#########6ac 67#####-########-####-#########6ac
    YYYY-MM-DDTHH:MM:SS: [vmfsCorrelator] 26764193007212us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-#########6ac 31#####-########-####-#########6ac

  • Following is the vmware.log of an affected VM showing vmware tools timeout and error writing log:
    YYYY-MM-DDTHH:MM:SS No(00) vmx - >>> Error writing log, 178 bytes discarded. Disk full?
    YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: Tools heartbeat timeout.
    YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: Running status rpc handler: 1 => 0.
    YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: Changing running status: 1 => 0.
    YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value 17560572 (last received 21s ago)
    YYYY-MM-DDTHH:MM:SS No(00) vcpu-7 - >>> Error writing log, 87 bytes discarded. Disk full?
    YYYY-MM-DDTHH:MM:SS No(00) vmx - >>> Error writing log, 106 bytes discarded. Disk full?
    YYYY-MM-DDTHH:MM:SS In(05) vmx - GuestRpc: GuestRpcResetVsockChannel: channel 1
    YYYY-MM-DDTHH:MM:SS In(05) vmx - GuestRpc: Closing channel 1 connection 3 
  • Kernel-level read-only remount errors on ext4 filesystems

Environment

  • vSphere 7.x
  • vSphere 8.x 
  • VMware Cloud Director 10.6.x

Cause

This issue occurs when the underlying storage datastore becomes inaccessible due to network or fabric issues, the Linux kernel remounts the filesystem as read-only to prevent corruption. This behavior is triggered when the ESXi host propagates storage I/O failures to the Guest OS.

Resolution

To resolve this issue the underlying storage connectivity problem must be resolved first. Then reboot the VM and remount the filesystem.

  1. Verify and restore connectivity to the underlying storage array or network fabric.
  2. After recovery of the storage connectivity and restoring of datastore/s to functional state, reboot the Linux VMs to restore original state of filesystem from read-only.
  3. If reboot does not fix, mount the filesystem manually using command:

    # mount -o remount /

  4. Verify the filesystem state using the mount or touch command.

If the issue persists, contact Broadcom Support.

  1.  

Additional Information

Powered by Wolken Software