Error: "Failed to start File System Check on /dev/vg_root_0/lv_root0" on Photon OS based virtual appliances

Products

• VMware Cloud Foundation
• VMware vCenter Server
• VMware SDDC Manager / VCF Installer
• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware Site Recovery Manager 8.x
• VMware Live Recovery
• VMware Cloud Director
• VMware vCenter Server 8.0
• VCF Operations
• VMware HCX

Issue/Introduction

• Following a system reboot, a power outage, datastore inaccessibility, or a similar disruption, the Photon OS-based virtual appliance boots into emergency mode
• The appliance fails to start normally and displays an error similar to the following:
  

  [FAILED] Failed to start File System Check on /dev/vg_root_0/lv_root0
  [DEPEND] Dependency failed for /sysroot.
  [DEPEND] Dependency failed for Initrd Root File System.
  [DEPEND] Dependency failed for Reload Configuration from the Real Root.

  
• The system is stuck in emergency mode with the following errors displayed:
  

  [FAILED] Failed to start File System #-####-####-###-##########
  [DEPEND] Dependency failed for /var/log/vmware.
  [DEPEND] Dependency failed for Local File Systems.
  You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"

  

• During emergency mode, a 'Dependency failed' error may be displayed for a specific mount point, such as the /storage/log directory:
  
  [FAILED] Failed to start File Syste= #-####-####-####-############
[DEPEND] Dependency failed for /storage/log.
[DEPEND] Dependency failed for Local File Systems.
You are in emergency Mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default Mode.
Give root password for Maintenance
(or press Control-D to continue):
  
  

• The following errors may be seen during boot
  

  dracut-initqueue[441]: Warning: Could not boot.
Starting Dracut Emergency Shell ...
Warning: /dev/vg_root_0/lv_root_0 does not exist

  Generating "/run/initramfs/rdsosreport. txt"

  Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport. txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

  Give root password for maintenance
or press Control-D to continue):



Environment

Photon OS-based appliance VMs

Cause

• This issue occurs when the virtual appliance experiences file inconsistencies as a result of a storage failure, power failure, or other crash.
• The virtual appliance may also enter emergency mode when it no longer has access to one or more of its disks. For example, if a hard disk is removed from the vCenter Server virtual appliance.

Resolution

To resolve this issue, scan and correct the filesystem by running the fsck command automatically (preferred) or manually.

 

Note: Before proceeding, take an offline snapshot of the affected virtual appliance.

Procedure:

1. Reboot the virtual appliance, and immediately after the OS starts, press 'e' to open the GNU GRUB Edit Menu. Locate the line that begins with the word linux

Option 1:
   
   At the end of the line, add fsck.repair=yes then press F10 to continue booting the appliance. This forces the filesystem to check and auto-resolve disk issues. The appliance silently reboots several times to fix issues as needed.
   
   
   
   Option 2:
   
   At the end of the line, add systemd.unit=emergency.target then press F10 to continue booting the appliance.
   
   
   
   
   1. 1. Find the affected filesystem:
         

         systemctl status systemd-fsck-root

         
         
         
      2. Run the fsck command against the mount point that has the issue:
         

         fsck -y /dev/<mount>

         Note: The "-y" switch fixes the nodes automatically. Replace <mount> with the mount point experiencing the issue.
         

         fsck -y /dev/mapper/vg_root_0-lv_root_0

         Alternatively, manually run a filesystem check and repair using the command below (the partition is just an example)
         

         e2fsck -y /dev/vg_root_0/lv_root_0

      3. Power off the virtual appliance:
         

         shutdown -h now

      4. Power on the virtual appliance.
         
         If the above steps do not recover the vCenter Server, restore the VM from File-Based backup following steps from  Restore vCenter Server from a File-Based Backup