Manual filesystem check for Aria Operations 8.x and above (Formerly vRealize Operations)
search cancel

Manual filesystem check for Aria Operations 8.x and above (Formerly vRealize Operations)

book

Article ID: 326392

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite) VCF Operations

Issue/Introduction

Note: This article is not suitable for vRealize Operations 6.x and 7.x. Aria Operations 8.x and above are running on PhotonOS, vRealize Operations 6.x and 7.x appliances run on SLES11.

Symptoms:
  • After rebooting or booting up a VCF Operations 8.x appliance, it requests you to log in as root in emergency mode.
  • You may see the error below for /dev/data/core in the VM console 

    [ SKIP ] Ordering cycle found, skipping Login Prompts
    [FAILED] Failed to start File System Check on /dev/data/core.
    [DEPEND] Dependency failed for /storage/core.
    [DEPEND] Dependency failed for Local File Systems.
    You are in emergency mode. After logging in, type "journalctl -xb" to view
    system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
    to boot into default mode.
    Give root password for maintenance
    (or press Control-D to continue):

 

  • Or an error similar to below:
    [FAILED] Failed to start File System Check on /dev/dis...uuid/XXXXXX-XXXX-
See 'systemctl status systemd-fsck-root.service' for details.

 

  • Or an error as below:

     [FAILED] Failed to start File System check on /dev/disk/by-label/root.
     [DEPEND] Dependency failed for /sysroot.
     [DEPEND] Dependency failed for Initrd Root File System.
     [DEPEND] Dependency failed for Mounts Configured in the Real Root.

 

     Generating "/run/initramfs/rdsosreport.txt"
     
     Entering emrgency mode. Exit the shell to continue.
     Type "journalctl" to view system logs.
     You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report.
     
     Press Enter for maintenance
     (or press Control-D to continue):

 

Environment

VMware Aria Operations 8.x
VCF Operations 9.0.x

Cause

This issue can occur when there is a storage failure, or power failure, resulting in filesystem inconsistencies.

Resolution

There are two types of partitions on the appliance that may require manual filesystem checks:

  • Standard partitions formatted as ext3 filesystems used for /boot and / (root)
    These partitions are identified as /dev/sda2 (/boot) and /dev/sda4 (/)

  • LVM partitions formatted as ext3 filesystems used for /storage/db, /storage/log, and /storage/core
    The LVM partitions will consist of one or more /dev/sdX partition(s). LVM manages these partitions, and you cannot perform e2fsck on these partitions directly, rather you must perform the filesystem check on the LVM volume itself. The LVM volumes are identified as /dev/mapper/data-log, /dev/mapper/data-db, and /dev/mapper/data-core.

To perform e2fsck on the two standard partitions, appliance must be booted into single user mode in read-only (ro) mode. To perform e2fsck on the three LVM partitions, appliance must be booted into single user mode in read-write (rw) mode to allow device node creation on root filesystem.

The procedure below can be performed in full, or on individual filesystems on an as-needed basis

 

Manual filesystem check on standard partitions (/ and /boot)

  1. Log into the vRealize/Aria Operations admin UI as the local admin user. 
  2. Select the desired node and click Take Node Offline/Online.

    Note: If single node cluster, or non-HA/CA cluster, click Take Offline under Cluster Status.
     
  3. Take a snapshot of the affected node(s) before proceeding. How to take a Snapshot of VMware Aria Operations
  4. In the vSphere Client, open the console of the desired node.
  5. With the console open, restart or power on the virtual machine.
  6. Type the letter e to go to the GNU GRUB edit menu.

    Notes:
    • Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
    • The cursor appears at the end of a line of boot options near the bottom of the display.
    • If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
       
  7. When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.
  8. Add a space, then type ro init=/bin/bash which adds another option to the line.

    Use 'ro' to ensure that filesystems are booted in read-only mode, as this is required to run filesystem checks on root and boot filesystems

    Note: When a filesystem is in read only mode, and you are unable to enter single user mode by the above means, run through steps 5-7 of the above section, and add the following boot option to force fsck:

    fsck.mode=force fsck.repair=yes

     
  9. Press F10.

    The virtual appliance starts in single-user mode.
     
  10. Type e2fsck -y /dev/sda2 to initiate filesystem check on boot (/boot) filesystem.
  11. Type e2fsck -y /dev/sda4 to initiate filesystem check on root (/) filesystem.
  12. When commands above are complete. Reboot with reboot -f to reboot to single user mode in read-write mode.
  13. If you also need to perform filesystem check on LVM filesystems, you can go directly to step 6 on procedure below. Please be aware that the GRUB menu will not remain on screen for very long.

 

Manual filesystem check on LVM partitions (/storage/db, /storage/log, and /storage/core)

  1. Log into the vRealize/Aria Operations admin UI as the local admin user. 
  2. Select the desired node and click Take Node Offline/Online.

    Note: If single node cluster, or non-HA/CA cluster, click Take Offline under Cluster Status.
     
  3. Take a snapshot of the affected node(s) before proceeding. How to take a Snapshot of VMware Aria Operations
  4. In the vSphere Client, open the console of the desired node.
  5. With the console open, restart or power on the virtual machine.
  6. Type the letter e to go to the GNU GRUB edit menu.

    Notes:
    • Press the up and down arrow keys even if the option appears to already be selected. Otherwise, the machine continues to boot, and you have to start over.
    • The cursor appears at the end of a line of boot options near the bottom of the display.
    • If you cannot reach the boot menu before it disappears, enable Force BIOS setup in the Virtual Machine's Settings > VM Options > Boot Options and reboot.
       
  7. When the GRUB loader menu appears, immediately use the up and down arrow keys to navigate to the end of the line that starts with Photon OS or Linux for new 8.x deployments.
  8. Add a space, then type rw init=/bin/bash which adds another option to the line.

    Use 'rw' to ensure that filesystems are booted in read-write mode

    Note: When a filesystem is in read only mode, and you are unable to enter single user mode by the above means, run through steps 5-7 of the above section, and add the following boot option to force fsck:

    fsck.mode=force fsck.repair=yes

     
  9. Press F10.

    The virtual appliance starts in single-user mode.
     
  10. Activate LVMs using commands below:

    lvchange -ay /dev/data/core
    lvchange -ay /dev/data/db
    lvchange -ay /dev/data/log

    Ignore the warnings regarding lvmetad service. This service does not run in single user mode.
     
  11. Type lvscan to ensure that all filesystems are ACTIVE.

    Example output:
    ACTIVE '/dev/data/core' [20.00 GiB] inherit
    ACTIVE '/dev/data/log' [20.00 GiB] inherit
    ACTIVE '/dev/data/db' [<40.00 GiB] inherit
     
  12. Type vgscan --mknodes to create device nodes for /dev/mapper devices.
  13. Run filesystem checks on all filesystems using commands below.

    e2fsck -y /dev/mapper/data-core
    e2fsck -y /dev/mapper/data-db
    e2fsck -y /dev/mapper/data-log

    Note: Do not issue all commands simultaneously. Filesystem check on data-db may take a long time, depending on the size of this filesystem
     
  14. When all 3 commands are complete. Reboot with reboot -f


Note: When a filesystem is in read only mode, and you are unable to enter single user mode by any of the above means, run through steps 5-7 of the above section, and add the following boot option to force fsck:

fsck.mode=force fsck.repair=yes

Additional Information

You may not need to run filesystem checks on all filesystems. When you see a failed to check filesystems message, it will indicate which filesystem that needs to be checked manually.
 

vRealize/Aria Operations has 5 filesystems that may need to be checked:

/dev/sda2 (/boot)
/dev/sda4 (/)
/dev/mapper/data-core
/dev/mapper/data-db
/dev/mapper/data-log

Powered by Wolken Software