Encryption Management Server fails to boot because of a corrupt file system

Products

• Encryption Management Server
• Gateway Email Encryption

Issue/Introduction

Encryption Management server fails to boot. At the console it displays errors relating to a corrupt file system.

Error messages like these appear in the console:

Checking filesystems
/ contains a file system with errors, check forced.

Corrupt File System Error - UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

Environment

Symantec Encryption Management Server 3.3.2 MP13 and above.

Cause

File system corruption is almost always the result of hardware failure. Reasons include:

1. Unexpected and abrupt power loss.
2. Actual or imminent disk failure resulting in a degradation of a RAID volume. Note that Encryption Management Server has no visibility of the health of the underlying RAID storage. You need to check the health of the RAID volume at system startup using the hardware vendor's ROM based RAID configuration utility or by booting with the hardware vendor's configuration media. Generally, the hardware vendor's RAID utility will warn you not only if a disk has already failed but if a disk is about to fail.
3. Imminent failure of a disk controller.

If Encryption Management Server is running as a VMWare ESXi virtual machine, the file system may be started in read only mode. Please see article TECH149087 for further details.

Resolution

The file system can often be repaired using the fsck file system check utility. However, there are no guarantees that this will be successful.

Method 1 - boot from hard drive

• Power on or restart Encryption Management Server.
• During the 3 second boot countdown process, press any key. Note that the version of the kernel will vary depending which release of Encryption Management Server you are using. Release 3.3.2 MP13 will have a kernel of 2.6.18-406.el5PAE. Release 3.4.2 MP3 will have a kernel of 2.6.32-754.15.3.el6.i686:

• In Encryption Management Server 3.4.2 MP3, the following screen appears. In Encryption Management Server 3.3.2 MP13, there is a second entry for Symantec Encryption Server-base which can be ignored:

• Press the letter e on the keyboard to edit the startup commands.
• On the next screen, press the cursor down key to select the line beginning kernel. Again, in Encryption Management Sever 3.3.2 MP13, the kernel line will differ:

• Press e on the keyboard to edit the kernel line.
• Navigate to the end of the line using the right cursor key and append a space and the text init=/bin/bash to the end of the line:

• Press the Enter key to return to the previous screen:

• Press b on the keyboard to boot and Encryption Management server will boot into the bash shell:

• Enter mount to show the mounted drives with a file system type of ext3 with Encryption Management Server 3.3.2 MP13 or ext4 with Encryption Management Server 3.4 and above. The name of the mounted drive will vary. For example, you may see /dev/sda2:

• Enter the following command to try to repair the file system. For example, enter this to repair /dev/sda2:
  /sbin/fsck -y /dev/sda2

• Here is an example where the file system did not need to be repaired:

• The file system repair may take 30 minutes or more.
• After the file system repair is completed, enter the command reboot -f to attempt an immediate but clean reboot of Encryption Management Server. Alternatively, enter the command poweroff -f to shutdown the system cleanly.

 

Method 2 - boot from ISO

• Boot from the Encryption Management Server installation ISO image. You may need to change the boot order in BIOS in order to ensure that the system boots from the ISO.
• At the setup screen enter the command noautopart and press the Enter key. The system will boot from ISO:

• At the Partitioning Type screen, press ALT+F2 on the keyboard to take you to the anaconda prompt:

• Enter the command fdisk -l to list the available partitions:

• Partition 2 is the largest partition. It may be called /dev/sda2. For example, enter fsck -y /dev/sda2 to repair /dev/sda2. Here is an example where the file system did not need to be repaired:

• The file system repair may take 30 minutes or more.
• After the file system repair is completed, enter the command poweroff to power off the system cleanly or the command reboot to reboot the system. Note however, that you may need to change the boot order to prevent the system booting from the ISO image again.

Additional Information

If you are running into a "grub>" prompt and will not even get beyond this, please reach out to Symantec Encryption Support for further guidance.