Manual filesystem check for vRealize/Aria Operations 8.x
search cancel

Manual filesystem check for vRealize/Aria Operations 8.x

book

Article ID: 326392

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

The purpose of this article is to provide instructions on how to perform manual filesystem checks on vRealize/Aria Operations 8.x Appliances. 

This article is not suitable for vRealize Operations 6.x and 7.x

Symptoms:
  • After rebooting or booting up a vRealize/Aria Operations 8.x appliance, it requests you to log in as root in emergency mode.
  • The appliance fails to start, and you may see an error similar to:

    [FAILED] Failed to start File System Check on /dev/dis...uuid/895462-1567-
    See 'systemctl status systemd-fsck-root.service' for details.


Environment

VMware vRealize Operations 8.x

Cause

This issue can occur when there is a storage failure, or power failure, resulting in filesystem inconsistencies.

Resolution

  1. Log into the vRealize/Aria Operations admin UI as the local admin user. 
  2. Select the desired node and click Take Node Offline/Online.

    Note: If single node cluster, or non-HA/CA cluster, click Take Offline under Cluster Status.
     
  3. Take a snapshot of the affected node(s) before proceeding. How to take a Snapshot of VMware Aria Operations
  4. In the vSphere Client, open the console of the desired node.
  5. With the console open, restart or power on the virtual machine.
  6. Type the letter e to go to the GNU GRUB edit menu.

    Notes:
    • Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
    • The cursor appears at the end of a line of boot options near the bottom of the display.
    • If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
       
  7. When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.
  8. Add a space, then type ro init=/bin/bash which adds another option to the line.

    Use 'ro' to ensure that filesystems are booted in read-only mode, as this is required to run filesystem checks on root and boot filesystems
     
  9. Press F10.

    The virtual appliance starts in single-user mode.
     
  10. Type e2fsck -y /dev/sda2 to initiate filesystem check on boot (/boot) filesystem.
  11. Type e2fsck -y /dev/sda4 to initiate filesystem check on root (/) filesystem.
  12. When commands above are complete. Reboot with reboot -f to reboot to single user mode in read-write mode.
  13. Type the letter e to go to the GNU GRUB edit menu.

    Notes:
    • Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
    • The cursor appears at the end of a line of boot options near the bottom of the display.
    • If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
       
  14. When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.
  15. Add a space, then type rw init=/bin/bash which adds another option to the line.

    Use 'rw' to ensure that filesystems are booted in read-write mode
     
  16. Press F10.

    The virtual appliance starts in single-user mode.
     
  17. Activate LVMs using commands below:

    lvchange -ay /dev/data/core
    lvchange -ay /dev/data/db
    lvchange -ay /dev/data/log

    Ignore the warnings regarding lvmetad service. This service does not run in single user mode.
     
  18. Type lvscan to ensure that all filesystems are ACTIVE.

    Example output:
    ACTIVE '/dev/data/core' [20.00 GiB] inherit
    ACTIVE '/dev/data/log' [20.00 GiB] inherit
    ACTIVE '/dev/data/db' [<40.00 GiB] inherit
     
  19. Type vgscan --mknodes to create device nodes.
  20. Run filesystem checks on all filesystems using commands below.

    e2fsck -y /dev/mapper/data-core
    e2fsck -y /dev/mapper/data-db
    e2fsck -y /dev/mapper/data-log

    Note: Do not issue all commands simultaneously. Filesystem check on data-db may take a long time, depending on the size of this filesystem
     
  21. When all 3 commands are complete. Reboot with reboot -f


Additional Information

You may not need to run filesystem checks on all filesystems. When you see the failed to check filesystems message, it will indicate which filesystem that needs to be checked manually.
 

vRealize/Aria Operations have 5 filesystems that may need to be checked:

/dev/sda2 (/boot)
/dev/sda4 (/)
/dev/mapper/data-core
/dev/mapper/data-db
/dev/mapper/data-log


Powered by Wolken Software