Manually running a filesystem check on a vCenter server
search cancel

Manually running a filesystem check on a vCenter server

book

Article ID: 338155

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

There are a number of reasons this can happen, but most commonly it is due to an underlying storage condition that causes the vCenter server to crash. Upon restart the services will not start, and some commands will fail to execute. 

On reboot of the vCenter appliance watching the screen will usually report the failing disk in the startup sequence.

If you see the below when attempting a service-control --status  command this can indicate filesystem inconsistency on the /storage/log partition:

root@VC# [ ~ ]# service-control --start --all
Traceback (most recent call last):
  File "/usr/bin/service-control", line 192, in <module>
    setupLogging('service-control', logMechanism='file', logDir=logDir)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 129, in __init__
    self._initLogFormat(program, level, logDir=logDir)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 155, in _initLogFormat
    create_dir(logDir)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 319, in create_dir
    os.makedirs(path)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/vmware/cloudvm'

Environment

VMware vCenter

Cause

Filesystem inconsistency. 

Resolution

Note: Ensure to take a snapshot or backup of vCenter Server Appliance before proceeding.
  1. Reboot vCenter Server Appliance, and immediately after the OS starts, press the key e to enter the GNU GRUB Edit Menu. 
  2. Locate the line that begins with the word linux. 
  3. Append the following to the end of that line:
    systemd.unit=emergency.target
    The line should look like the following screenshot

    • Sometimes the VCSA console will refuse to get into emergency mode and will be stuck at the screen below.
    • In this case, enter the line below instead of "systemd.unit=emergency.target"
      fsck.repair=yes 
  4. Press F10 to continue booting.
    Note: Enter the root password when prompted, it may have to be entered twice. 
  5. Change the BASH Shell by entering the command Shell. 
  6. Run this command df to list the available file systems:
    root[ ~ ]# df
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 8205180 0 8205180 0% /dev
tmpfs 8211976 8 8211968 1% /dev/shm
tmpfs 8211976 712 8211264 1% /run
tmpfs 8211976 0 8211976 0% /sys/fs/cgroup
/dev/sda3 11089144 4078128 6441060 39% /
tmpfs 8211976 10760 8201216 1% /tmp
/dev/mapper/netdump_vg-netdump 1007640 1304 954320 1% /storage/netdump
/dev/mapper/dblog_vg-dblog 15342576 104144 14452412 1% /storage/dblog
/dev/mapper/db_vg-db 10181944 98716 9559352 2% /storage/db
/dev/mapper/autodeploy_vg-autodeploy 10181944 23168 9634900 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder 10181944 23172 9634896 1% /storage/imagebuilder
/dev/mapper/seat_vg-seat 25663804 72344 24281152 1% /storage/seat
/dev/mapper/updatemgr_vg-updatemgr 103073056 64728 97765860 1% /storage/updatemgr
/dev/mapper/log_vg-log 10181944 67664 9590404 1% /storage/log
/dev/mapper/core_vg-core 51466876 53200 48792648 1% /storage/core
/dev/sda1 122835 27169 89113 24% /boot
  7. Run this command to perform a consistency check on the specified file system:
    #e2fsck -y filesystem
    For example:
    e2fsck -y /dev/sda1  
  8. Reboot vCenter Server Appliance by running this command:
    reboot -f

Additional Information

Refer to article below for more information:

Running the command in emergency/ GRUB mode, can help fix all orphaned inodes fsck -y /dev/mapper/*

The below failure can also mean the /storage/log volume isn't mounted. See KB: Unable to launch the vCenter UI 

root@VC# [ ~ ]# service-control --start --all
Traceback (most recent call last):
  File "/usr/bin/service-control", line 192, in <module>
    setupLogging('service-control', logMechanism='file', logDir=logDir)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 129, in __init__
    self._initLogFormat(program, level, logDir=logDir)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 155, in _initLogFormat
    create_dir(logDir)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 319, in create_dir
    os.makedirs(path)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/vmware/cloudvm'

Powered by Wolken Software