Error: fsck fails during vCenter server virtual machine startup

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• During the reboot of the vCenter VM, the sytem drops into an emergency mode with the FSCK (File System Check) phase and it prevents the vCenter from fully booting up.
• The error message indicates that a manual file system check (fsck) is required on the VMware Photon OS environment.

Environment

VMware vCenter Server

Cause

• Any scenario where the vCenter VM is abruptly shutdown due to the power outage or manually powered off (without Guest OS shutdown), it leaves the file system of the vCenter into an inconsistent state.

• If the storage array hosting datastore of vCenter VM gets into APD/PDL state, this causes vCenter to queue I/O operations to an unresponsive datastore leading to file system corruption in the guest OS of the vCenter VM.

Resolution

To resolve this issue, the file system inconsistencies must be fixed by manually executing fsck.

• Prerequisites before following the  steps below.

  • root password of the vCenter Server Appliance.

  • Access the vCenter VM console directly through the ESXi host web client (since vCenter is offline).

  • Ensure the underlying storage issue and power outage is addressed before proceeding with the steps below.

1. Take a snapshot of the vCenter Server Virtual Machine (VM) by referring to Snapshot Best practices for vCenter Server Virtual Machines.
   
   
2. Restart the Virtual Machine running VMware Photon OS. When the splash screen appears, quickly press the letter e to enter the Grand Unified Bootloader (GRUB) edit menu.
   
   
   • Note: The letter e must be pressed quickly as the operating system reboots rapidly. To ensure that enough time is provided during the boot to press 'e' key, the Boot Delay setting can be changed to 10000 milliseconds (10 seconds). Refer to this article for more details Delay the Boot Sequence
     
     
3. In the GNU GRUB edit menu, navigate to the end of the line that begins with linux, add a space, and append the following code exactly: systemd.unit=emergency.target


   
   
4. Press F10.
   
   
5. In the bash shell, execute one of the following commands to fix the file system errors, depending on whether sda1 or sda2 represents the root file system:
   
   e2fsck -y /dev/sda1 or e2fsck -y /dev/sda2
   
   
6. Restart the Virtual Machine.