Root partition on an ESX host becomes read only

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Root partition on an ESX host becomes read only.
• You are unable to manage your ESX host.
• The ESX host has a status of Not Responding or Disconnected in VMware vCenter Server.
• You cannot power up virtual machines or work with their consoles.
• The virtual machines appear to still be responsive and operational. You can access them via ping or RDP.
  
• You cannot start the management agents from the terminal.
• You are unable to perform touch <file> on any file in the root partition.
• If you try to perform touch <file>, you see this error:

touch: cannot touch <file>: "Read-only file system"

Environment

VMware ESX Server 3.5.x
VMware ESX 4.0.x
VMware ESX 4.1.x
VMware ESX Server 3.0.x

Resolution

This issue can occur due to:

• Damaged or corrupted file system.
• Hard drive or controller failure.
• The file system is indeed full.

The root partition is remounted in Read-Only access mode to prevent further problems from being committed to the file system or disk. To correct this issue, review the /var/log/vmkernel and /var/log/messages files, and examine if there any ext3 file system journal errors in the log. If journal errors exist, it suggests file system corruption that needs to be repaired.

Note: Resolving this issue requires a downtime of the ESX host. Ensure that you plan for a proper maintenance window including a possible ESX host rebuild.

To check for and repair file system corruption, run the filesystem check utility.

To run a filesystem check:

1. Log in to the Service Console of your ESX host as the root user.
2. Reboot the ESX host by typing reboot.
3. When the GRUB screen appears, press the Space bar to stop the server from automatically booting into VMware ESX .
4. Use the arrow keys to select Service Console only (troubleshooting mode).
5. Press the a key to modify the kernel arguments (boot options) and press the Space bar and type the word single.
6. Press Enter. The server continues to boot into single-user mode.
7. When presented with a bash prompt such as sh-2.05b#, type fdisk -l and make a note of the affected partition.
8. Run fsck -a /dev/xxx
   
   Where xxx is the affected partition name (For example sdb5).
   
   
9. After the disk check completes, reboot the server into the normal run-level.

Additional Information

If the disk check operation fails or if the EXT3 journal becomes corrupt on the root file system, contact your hardware vendor for assistance as the corruption may be caused by hardware failures.

If the issue persists, you may have to rebuild the ESX host.