"fsck failed" error when starting vCenter Server Appliance

Products

• VMware Live Recovery
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Cannot start the VMware vCenter Server Appliance
• Errors may appear as:
  
  /dev/mapper/log_vg-log: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY

fsck failed. Please repair manually and reboot. The root file system is currently mounted read-only. To remount it read-write do:
bash# mount -n -o remount,rw /
Attention: Only CONTROL-D will reboot the system in this
maintanance mode. Shutdown or reboot will not work.
Give root password for maintenance
(or type Control-D to continue):

Cause

This issue occurs when the appliance is configured to perform a pre-disk check every time it starts. In this case, the routine disk check discovers a serious issue that needs operator intervention.

Resolution

Note: Ensure to create a backup or snapshot of the appliance before proceeding.

 

To resolve this issue, check the file system for errors and repair as appropriate.

Option 1:

1. Power on the vCenter Server Appliance.
   
   Note: In vCenter Server Appliance 6.0, it is required to enable BASH shell.
   
   
2. Open a console to the appliance and enter the root credentials.
   
   
3. Run this command to check for disk errors:
   
   Caution: When clicking the 'y' key to fix an error, the 'e2fsck' command corrects the state of the disk it is currently working.
   
   e2fsck -y /dev/sda2
   
   
4. Run this command to remount the file system:
   
   mount -n -o remount,rw /
   
   
5. Press CTRL+D to exit maintenance mode and complete the starting process.

Option 2:

1. Press Control+D or reboot the vCSA using the vSphere Client.

2. When the GRUB bootloader appears, press Spacebar to disable autoboot.

3. Select 'p' and enter the root password

4. Type 'e' to edit the boot commands.

5. Scroll to the second line displaying the kernel boot parameters.
   
   Find this line:
   kernel /vmlinuz-3.0.101-0.35-default

6.  

   Type 'e' to edit the boot command.

7. Append the following to the kernel boot options:
   
   init=/bin/bash 

8. Press Enter. The GRUB menu re-appears.

9. Type 'b' to start the boot process. The system boots to a temporary BASH shell.

10. Set the default shell to BASH by running this command:
    
    chsh -s /bin/bash root

11. Reboot the vCSA using the vSphere Client.

12. Enter the root password.

13. At the repair filesystem prompt, type this command:
    
    fsck /dev/mapper/log_vg-log

14. Once complete, reboot the vCSA.

Enabling BASH shell in vCenter Server Appliance 6.0

 

vCenter Server Appliance 6.0 does not boot to the BASH shell by default.

 

To enable the BASH shell:

1. In the GRUB boot loader press p.
   
   
2. Enter the root credentials when prompted.
   
   
3. Select VMware vCenter Server Appliance and press e.
   
   
4. Select the line that begins with kernel and press e.
   
   
5. At the end of the line, add the below line and press Enter. This line should start with nousb audit=1.
   
   init=/bin/bash
   
   
6. Press b.