"Failed to start file system check on /dev/disk..." error on Photon OS based virtual appliances
search cancel

"Failed to start file system check on /dev/disk..." error on Photon OS based virtual appliances

book

Article ID: 326323

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware vCenter Server VMware SDDC Manager VMware Aria Suite

Issue/Introduction

  • After rebooting vCenter Server the system enters emergency mode.
  • The appliance fails to start and there is an error similar to:

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.


Environment

  • VMware vCenter Server 6.x
  • VMware vCenter Server 7.x
  • VMware vCenter Server 8.x
  • VMware SDDC Manager 4.x
  • VMware SDDC Manager 5.x 
  • VMware Aria Suite Lifecycle 8.x
  • VMware Aria Automation 8.x
  • VMware Aria Automation Orchestrator 8.x
  • VMware Identity Manager 3.3.7

Cause

This issue occurs when the virtual appliances experience file inconsistencies after being forcefully halted as the result of a storage failure, power failure, or software crash.

Resolution

To resolve this issue, scan and correct the filesystem by running the fsck command automatically (preferred) or manually.
 
Note: Before proceeding, take a snapshot of the affected virtual appliance.
  1. Reboot the virtual appliance, and immediately after the OS starts, press 'e' to open the GNU GRUB Edit Menu.
  2. Locate the line that begins with the word linux.

Option 1
At the end of the line, add fsck.repair=yes then press F10 to continue booting the appliance. This will force the default filesystem check to auto-resolve any issues. The appliance may silently reboot several times to fix needed issues.

Option 2
At the end of the line, add systemd.unit=emergency.target then press F10 to continue booting the appliance.

1. Find the filesystems by running the following commands:

  • $ /bin/sh
  • $ /bin/mount
  • $ blkid

2. Run this command against the mount point that has issue:

$ e2fsck -y /dev/<mount>

The -y switch will fix the nodes automatically. Replace <mount> with the mount point experiencing the issue. Ex: $ e2fsck -y /dev/sda3

3. Power OFF the virtual appliance.

4. Power ON the virtual appliance.

Additional Information

By running pam_tally2 -u root via SSH on each node after we were able to get logged in, we determined that the Lockouts were coming from Solar Winds. 

service loginsight stop; umount /storage/core; umount /storage/var
e2fsck -y /dev/mapper/data-var
e2fsck -y /dev/mapper/data-core

Powered by Wolken Software