Photon OS based Virtual appliance fails to boot with error: "Failed to start File System Check on /dev/vg_root_0/lv_root

Products

VMware Cloud Foundation VMware vCenter Server VMware SDDC Manager / VCF Installer VCF Operations/Automation (formerly VMware Aria Suite) VMware Site Recovery Manager 8.x VMware Live Recovery VMware Cloud Director VMware vCenter Server 8.0 VCF Operations VMware HCX

Issue/Introduction

Following a system reboot, a power outage, datastore inaccessibility, or a similar disruption, the Photon OS-based virtual appliance boots into emergency mode

The appliance fails to start normally and displays an error similar to the following:

[FAILED] Failed to start File System Check on /dev/vg_root_0/lv_root_0
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File System.
[DEPEND] Dependency failed for Reload Configuration from the Real Root.

The system is stuck in emergency mode with the following errors displayed:

[FAILED] Failed to start File System #-####-####-###-##########
[DEPEND] Dependency failed for /var/log/vmware.
[DEPEND] Dependency failed for Local File Systems.
You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"

During emergency mode, a 'Dependency failed' error may be displayed for a specific mount point, such as the /storage/log directory:

[FAILED] Failed to start File Syste= #-####-####-####-############
[DEPEND] Dependency failed for /storage/log.
[DEPEND] Dependency failed for Local File Systems.
You are in emergency Mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default Mode.
Give root password for Maintenance
(or press Control-D to continue):

The following errors may be seen during boot

dracut-initqueue[441]: Warning: Could not boot.
Starting Dracut Emergency Shell ...
Warning: /dev/vg_root_0/lv_root_0 does not exist

Generating "/run/initramfs/rdsosreport. txt"

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport. txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

Give root password for maintenance
or press Control-D to continue):

Environment

Photon OS-based appliance VMs
VMware vCenter Server
VCF Operations 9.x

Cause

This issue occurs when the virtual appliance experiences file inconsistencies as a result of a storage failure, power failure, or other crash.
The virtual appliance may also enter emergency mode when it no longer has access to one or more of its disks. For example, if a hard disk is removed from the vCenter Server virtual appliance.

Resolution

To resolve this issue, scan and correct the filesystem by running the fsck command automatically (preferred) or manually.

Note: Before proceeding, take an offline snapshot of the affected virtual appliance.

Procedure:

Reboot the affected virtual appliance, and immediately after the OS starts, press 'e' to open the GNU GRUB Edit Menu. Locate the line that begins with the word linuxOption 1:

At the end of the line, add fsck.repair=yes then press F10 to continue booting the appliance. This forces the filesystem to check and auto-resolve disk issues. The appliance silently reboots several times to fix issues as needed.

Option 2:

At the end of the line, add systemd.unit=emergency.target then press F10 to continue booting the appliance.
1. 1. Find the affected filesystem:
```
systemctl status systemd-fsck-root
```
  2. Run the fsck command against the mount point that has the issue:
```
fsck -y /dev/<mount>
```
    Note: The "-y" switch fixes the nodes automatically. Replace <mount> with the mount point experiencing the issue.
```
fsck -y /dev/mapper/vg_root_0-lv_root_0
```
    Alternatively, manually run a filesystem check and repair using the command below (the partition is just an example)
```
e2fsck -y /dev/vg_root_0/lv_root_0
```
  3. Power off the virtual appliance:
```
shutdown -h now
```
  4. Power on the virtual appliance.
  5. If the above steps do not recover the affected appliance, restore the VM from File-Based backup.
    
    Component-Specific Restore Reference: To execute Step e, locate the affected appliance and follow the corresponding official procedure.
    
    For vCenter: Restore vCenter from a File-Based Backup
    For VCF Operations: Image-Based Restore of VCF Operations
    SDDC Manager: Restore SDDC Manager
    VCF Automation: File-Based Restore of VCF Automation

Additional Information

Photon OSベースの仮想アプライアンスで"Failed to start File System Check on /dev/vg_root_0/lv_root_0"というエラーが発生