"Failed to start file system check on /dev/disk..." error on Photon OS based virtual appliances

Products

• VMware Cloud Foundation
• VMware vCenter Server
• VMware SDDC Manager
• VMware Aria Suite
• VMware Site Recovery Manager 8.x
• VMware Live Recovery
• VMware Cloud Director

Issue/Introduction

Symptoms:

• After rebooting, power outage, datastore inaccessibility or other issues, the Photon OS based virtual appliance enters emergency mode.
• The appliance fails to start and there is an error similar to:

[FAILED] Failed to start File System Check on /dev/disk...uuid/#####-### See 'systemctl status systemd-fsck-root.service' for details.
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File System.
[DEPEND] Dependency failed for Reload Configuration from the Real Root.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

• VMware vCenter Server 6.x
• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware SDDC Manager 4.x
• VMware SDDC Manager 5.x
• VMware Aria Suite Lifecycle 8.x
• VMware Aria Automation 8.x
• VMware Aria Automation Orchestrator 8.x
• VMware Identity Manager 3.3.7
• VMware Live Site Recovery 9.x
• VMware Site Recovery 8.8
• VMware vSphere Replication 8.x
• VMware vSphere Replication 9.x
• VMware Cloud Director 10.X

Cause

This issue occurs when the virtual appliance experiences file inconsistencies after being forcefully halted as the result of a storage failure, power failure, or other crash.

Resolution

To resolve this issue, scan and correct the filesystem by running the fsck command automatically (preferred) or manually.

 

Note: Before proceeding, take a snapshot of the affected virtual appliance.

1. Reboot the virtual appliance, and immediately after the OS starts, press 'e' to open the GNU GRUB Edit Menu.
2. Locate the line that begins with the word linux.

Option 1:

At the end of the line, add fsck.repair=yes then press F10 to continue booting the appliance. This will force the filesystem to check and auto-resolve disk issues. The appliance may silently reboot several times to fix issues as needed.

Option 2:

At the end of the line, add systemd.unit=emergency.target then press F10 to continue booting the appliance.

1. Find the filesystems with the following commands:

• $ /bin/sh
• $ /bin/mount
• $ blkid

2. Run the e2fsck command against the mount point that has the issue:

$ e2fsck -y /dev/<mount>

Note: The -y switch will fix the nodes automatically. Replace <mount> with the mount point experiencing the issue. Ex: $ e2fsck -y /dev/sda3

3. Power OFF the virtual appliance.

4. Power ON the virtual appliance.