Aria Automation appliance failed to load the Photon OS and continously booting into emergency mode

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

After a recent vSAN disk failure Aria Automation appliance failed to load the Photon OS and continously booting into emergency mode.

[FAILED] Failed to start File System Check on /dev/data_vg/data
See 'systemctl status systemd-fsck@dev-data_vg-data.service' for details 
Dependency failed for /data 
Dependency failed for Local File Systems 

Environment

VMware Aria Automation 8.x

Cause

This issue occurs when a VM  is forcefully halted as a result of storage failure, power failure, or software stack crash, causing file inconsistencies.

Resolution

Caution: Before proceeding, take a snapshot or backup of the affected virtual appliance.

• Reboot the virtual appliance, and immediately after the OS starts, press " e " to open the GNU GRUB Edit Menu.
• Locate the line that begins with the word linux.
  At the end of the line, add " fsck.repair=yes " then press F10 to continue booting the appliance. This will force the default filesystem check to auto-resolve any issues.
• Power off the virtual appliance.
• Power on the virtual appliance.
  Depending on the number of affected node and the status of services we might need to execute " /opt/scripts/deploy.sh " 
• Validates the status of all the nodes:

  kubectl get nodes

• Run the following command to verify that all services are running:

  kubectl -n prelude get pods

Additional Information

You can also repair the file system without accessing the GRUB menu by entering the appliance in maintenance mode with the root password and then running the following command:

fsck -y /dev/dta_vg/data