Aria Automation appliance failed to load the Photon OS and continously booting into emergency mode

Products

• VMware Aria Suite

Issue/Introduction

After a recent vSAN disk failure Aria Automation appliance failed to load the Photon OS and continously booting into emergency mode. 

FAILED Failed to start File System Check on /dev/data_vg/data
See 'systemctl status systemd-fsck@dev-data_vg-data.service' for details 
Dependency failed for /data 
Dependency failed for Local File Systems 

 

Environment

3 nodes clustered  Aria Automation 8.x . One or more nodes in the cluster are constantly booting into emergency mode. 

Cause

This issue occurs when a VM  is forcefully halted as a result of storage failure, power failure, or software stack crash, causing file inconsistencies.

Resolution

Caution: Before proceeding, take a snapshot or backup of the affected virtual appliance.

• Reboot the virtual appliance, and immediately after the OS starts, press " e " to open the GNU GRUB Edit Menu.
• Locate the line that begins with the word linux.
  At the end of the line, add " fsck.repair=yes " then press F10 to continue booting the appliance. This will force the default filesystem check to auto-resolve any issues.
• Power off the virtual appliance.
• Power on the virtual appliance.
  Depending on the number of affected node and the status of services we might need to execute " /opt/scripts/deploy.sh " 
• Validates the status of all the nodes:
  kubectl get nodes   
• Run the following command to verify that all services are running:
  kubectl -n prelude get pods