Manual filesystem check for Aria Operations for Logs (Formerly vRealize Log Insight)
search cancel

Manual filesystem check for Aria Operations for Logs (Formerly vRealize Log Insight)

book

Article ID: 379277

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

How to perform manual filesystem checks on Aria Operations for Logs / vRealize Log Insight 8.x Appliances. 

This article is not suitable for vRealize Log Insight 7.x and older versions. Aria Operations for Logs / vRealize Log Insight 8.x onwards is running on PhotonOS, 7.x and older appliances runs on SLES11.

Symptoms:
  • After rebooting or booting up an Aria Operations for Logs / vRealize Log Insight 8.x appliance, it requests you to log in as root in emergency mode.
  • The appliance fails to start, and you may see an error similar to:
    [FAILED] Failed to start File System Check on /dev/dis...uuid/XXXXXX-XXXX-
See 'systemctl status systemd-fsck-root.service' for details.

Environment

VMware Aria Operations for Logs 8.x

VMware vRealize Log Insight 8.x

Cause

This issue can occur when there is a storage failure, or power failure, resulting in filesystem inconsistencies.

Resolution

There are two types of partitions on the appliance that may require manual filesystem checks:



  • Standard partitions formatted as ext3 filesystems used for /boot and / (root)
    These partitions are identified as /dev/sda2 (/boot) and /dev/sda4 (/)


  • LVM partitions formatted as ext3 filesystems used for /storage/core and /storage/var
    The LVM partitions will consist of one or more /dev/sdX partition(s). LVM manages these partitions, and you cannot perform e2fsck on these partitions directly, rather you must perform the filesystem check on the LVM volume itself. The LVM volumes are identified as /dev/mapper/data-core and /dev/mapper/data-var.


To perform e2fsck on the two standard partitions, appliance must be booted into single user mode in read-only (ro) mode. To perform e2fsck on the three LVM partitions, appliance must be booted into single user mode in read-write (rw) mode to allow device node creation on root filesystem.


The procedure below can be performed in full, or on individual filesystems on an as-needed basis


 


Manual filesystem check on standard partitions (/ and /boot)




  1. Log into the Aria Operations for Logs / vRealize Log Insight UI as the local admin user. 

  2. Navigate to Management > Cluster, click "Put this node into maintenance mode" (Pause icon).

  3. Take a snapshot of the affected node(s) before proceeding.

  4. In the vSphere Client, open the console of the desired node.

  5. With the console open, restart or power on the virtual machine.

  6. Type the letter e to go to the GNU GRUB edit menu.

    Notes:

    • Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.

    • The cursor appears at the end of a line of boot options near the bottom of the display.

    • If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
       



  7. When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.

  8. Add a space, then type ro init=/bin/bash which adds another option to the line.

    Use 'ro' to ensure that filesystems are booted in read-only mode, as this is required to run filesystem checks on root and boot filesystems
     

  9. Press F10.

    The virtual appliance starts in single-user mode.
     

  10. Type e2fsck -y /dev/sda2 to initiate filesystem check on boot (/boot) filesystem.

  11. Type e2fsck -y /dev/sda4 to initiate filesystem check on root (/) filesystem.

  12. When commands above are complete. Reboot with reboot -f to reboot to single user mode in read-write mode.

  13. If you also need to perform filesystem check on LVM filesystems, you can go directly to step 6 on procedure below. Please be aware that the GRUB menu will not remain on screen for very long.



 


Manual filesystem check on LVM partitions (/storage/var and /storage/core)




  1. Log into the Aria Operations for Logs / vRealize Log Insight UI as the local admin user. 

  2. Navigate to Management > Cluster, click "Put this node into maintenance mode" (Pause icon).

  3. Take a snapshot of the affected node(s) before proceeding.

  4. In the vSphere Client, open the console of the desired node.

  5. With the console open, restart or power on the virtual machine.

  6. Type the letter e to go to the GNU GRUB edit menu.

    Notes:

    • Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.

    • The cursor appears at the end of a line of boot options near the bottom of the display.

    • If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
       



  7. When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.

  8. Add a space, then type rw init=/bin/bash which adds another option to the line.

    Use 'rw' to ensure that filesystems are booted in read-write mode
     

  9. Press F10.

    The virtual appliance starts in single-user mode.
     

  10. Activate LVMs using commands below:

    lvchange -ay /dev/data/core
    lvchange -ay /dev/data/var

    Ignore the warnings regarding lvmetad service. This service does not run in single user mode.
     

  11. Type lvscan to ensure that all filesystems are ACTIVE.

    Example output:
    ACTIVE '/dev/data/core' [4.00 TiB] inherit
    ACTIVE '/dev/data/var' [20.00 GiB] inherit
     

  12. Type vgscan --mknodes to create device nodes for /dev/mapper devices.

  13. Run filesystem checks on all filesystems using commands below.

    e2fsck -c -y /dev/mapper/data-var
    e2fsck -c -y /dev/mapper/data-core

    Note: Do not issue all commands simultaneously. Filesystem check on data-core may take a long time, depending on the size of this filesystem. 
     

  14. When all 3 commands are complete. Reboot with reboot -f


Additional Information

You may not need to run filesystem checks on all filesystems. When you see a failed to check filesystems message, it will indicate which filesystem that needs to be checked manually.
 

Aria Operations for Logs/vRealize Log Insight have 4 filesystems that may need to be checked:

/dev/sda2 (/boot)
/dev/sda4 (/)
/dev/mapper/data-core
/dev/mapper/data-var

Powered by Wolken Software