"Failed to start file system check on /dev/disk..." error on Photon OS based virtual appliances

Products

• VMware Cloud Foundation
• VMware vCenter Server
• VMware SDDC Manager

Issue/Introduction

• After rebooting vCenter Server the system enters emergency mode.
• The appliance fails to start and there is an error similar to:
  

  [FAILED] Failed to start File System Check on /dev/dis...uuid/XXXXX-XXX-
  See 'systemctl status systemd-fsck-root.service' for details.
  [DEPEND] Dependency failed for /sysroot.
  [DEPEND] Dependency failed for Initrd Root File System.
  [DEPEND] Dependency failed for Reload Configuration from the Real Root.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment

Environment

• VMware vCenter Server 6.5.x
• VMware vCenter Server 6.7.x
• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware SDDC Manager 4.x
• VMware SDDC Manager 5.x 

Cause

This issue occurs when the vCenter Server experiences file inconsistencies after being forcefully halted as the result of a storage failure, power failure, or software crash.

 

Resolution

To resolve this issue, scan and correct the filesystem by running the fsck command automatically (preferred) or manually.

 

Note: Before proceeding, take a snapshot of the affected virtual appliance.

1. Reboot the virtual appliance, and immediately after the OS starts, press 'e' to open the GNU GRUB Edit Menu.
2. Locate the line that begins with the word linux.

Option 1
At the end of the line, add fsck.repair=yes then press F10 to continue booting the appliance. This will force the default filesystem check to auto-resolve any issues. The appliance may silently reboot several times to fix needed issues.

Option 2
At the end of the line, add systemd.unit=emergency.target then press F10 to continue booting the appliance.

1. Find the filesystems by running the following commands:

• $ /bin/sh
• $ /bin/mount
• $ blkid

2. Run this command against the mount point that has issue
$ e2fsck -y /dev/<mount>

The -y switch will fix the nodes automatically. Replace <mount> with the mount point experiencing the issue. Ex: $ e2fsck -y /dev/sda3

3. Power OFF the virtual appliance.

4. Power ON the virtual appliance.