Manual filesystem check for Aria Operations for Logs (Formerly vRealize Log Insight)
book
Article ID: 379277
calendar_today
Updated On:
Products
VMware Aria Suite
Issue/Introduction
How to perform manual filesystem checks on Aria Operations for Logs / vRealize Log Insight 8.x Appliances.
This article is not suitable for vRealize Log Insight 7.x and older versions. Aria Operations for Logs / vRealize Log Insight 8.x onwards is running on PhotonOS, 7.x and older appliances runs on SLES11.
Symptoms:
After rebooting or booting up an Aria Operations for Logs / vRealize Log Insight 8.x appliance, it requests you to log in as root in emergency mode.
The appliance fails to start, and you may see an error similar to:
[FAILED] Failed to start File System Check on /dev/dis...uuid/XXXXXX-XXXX-
See 'systemctl status systemd-fsck-root.service' for details.
Environment
VMware Aria Operations for Logs 8.x
VMware vRealize Log Insight 8.x
Cause
This issue can occur when there is a storage failure, or power failure, resulting in filesystem inconsistencies.
Resolution
There are two types of partitions on the appliance that may require manual filesystem checks:
Standard partitions formatted as ext3 filesystems used for /boot and / (root) These partitions are identified as /dev/sda2 (/boot) and /dev/sda4 (/)
LVM partitions formatted as ext3 filesystems used for /storage/core and /storage/var The LVM partitions will consist of one or more /dev/sdX partition(s). LVM manages these partitions, and you cannot perform e2fsck on these partitions directly, rather you must perform the filesystem check on the LVM volume itself. The LVM volumes are identified as /dev/mapper/data-core and /dev/mapper/data-var.
To perform e2fsck on the two standard partitions, appliance must be booted into single user mode in read-only (ro) mode. To perform e2fsck on the three LVM partitions, appliance must be booted into single user mode in read-write (rw) mode to allow device node creation on root filesystem.
The procedure below can be performed in full, or on individual filesystems on an as-needed basis
Manual filesystem check on standard partitions (/ and /boot)
Log into the Aria Operations for Logs / vRealize Log Insight UI as the local admin user.
Navigate to Management > Cluster, click "Put this node into maintenance mode" (Pause icon).
Take a snapshot of the affected node(s) before proceeding.
In the vSphere Client, open the console of the desired node.
With the console open, restart or power on the virtual machine.
Type the letter e to go to the GNU GRUB edit menu.
Notes:
Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
The cursor appears at the end of a line of boot options near the bottom of the display.
If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.
Add a space, then type ro init=/bin/bash which adds another option to the line.
Use 'ro' to ensure that filesystems are booted in read-only mode, as this is required to run filesystem checks on root and boot filesystems
Press F10.
The virtual appliance starts in single-user mode.
Type e2fsck -y /dev/sda2 to initiate filesystem check on boot (/boot) filesystem.
Type e2fsck -y /dev/sda4 to initiate filesystem check on root (/) filesystem.
When commands above are complete. Reboot with reboot -f to reboot to single user mode in read-write mode.
If you also need to perform filesystem check on LVM filesystems, you can go directly to step 6 on procedure below. Please be aware that the GRUB menu will not remain on screen for very long.
Manual filesystem check on LVM partitions (/storage/var and /storage/core)
Log into the Aria Operations for Logs / vRealize Log Insight UI as the local admin user.
Navigate to Management > Cluster, click "Put this node into maintenance mode" (Pause icon).
Take a snapshot of the affected node(s) before proceeding.
In the vSphere Client, open the console of the desired node.
With the console open, restart or power on the virtual machine.
Type the letter e to go to the GNU GRUB edit menu.
Notes:
Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
The cursor appears at the end of a line of boot options near the bottom of the display.
If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.
Add a space, then type rw init=/bin/bash which adds another option to the line.
Use 'rw' to ensure that filesystems are booted in read-write mode
Note: Do not issue all commands simultaneously. Filesystem check on data-core may take a long time, depending on the size of this filesystem.
When all 3 commands are complete. Reboot with reboot -f
Additional Information
You may not need to run filesystem checks on all filesystems. When you see a failed to check filesystems message, it will indicate which filesystem that needs to be checked manually.
Aria Operations for Logs/vRealize Log Insight have 4 filesystems that may need to be checked: