vCenter VAMI Alert: File system /storage/log is not writeable

Products

• VMware vCenter Server

Issue/Introduction

The vCenter VAMI displays a critical health alert indicating that the /storage/log file system is not writeable as below :

 

 

vCenter services status shows as below : 

# service-control -- status -- all

Traceback (most recent call last) :

  File "/usr/bin/service-control", line 192, in <module>

    setupLogging ('service-control', logMechaniam='file', logDir=logDir)

  File "/usr/lib/vmware/site-packages/cis/utils.py", line 149, in init

    self. setupFileLogging (program, level-level, logDir-logDir,

  File "/usr/lib/vmware/site-packages/cis/utils.py", line 184, in setupFileLogging

    fileHandler = logging. FileHandler (logFile)

  File "/usr/lib/python3.10/logging/ init .py", line 1169, in init

    StreamHandler. init (self, self. open () )

  File "/usr/lib/python3.10/logging/ init .py", line 1201, in open

     return open_func (self.baseFilename, self.mode,

OSError: [Errno 30] Read-only file system: "/var/log/vmware/cloudvm/service-control.log'

 

vCenter vm console shows the below :

 

Environment

vCenter 8.x

Cause

Potential causes for this issue include:

• File System Corruption: vCenter system files may be corrupted or stuck in a read-only state.

• Emergency Mode: The vCenter appliance may have booted into emergency mode.

• Storage Instability: There may be underlying, intermittent storage issues.

• Improper Shutdown: The vCenter server may have experienced an unexpected reboot or power loss.

Resolution

 

1. Take snapshot of the vCenter VM. ( Offline snapshots of all vCenters if in linked mode ) 

2. Reboot the vCenter appliance.

3. Interrupt the boot process by pressing the "e" key to enter the GRUB menu.

4. Append the text fsck.repair=yes to the kernel boot line as per https://knowledge.broadcom.com/external/article/326323

5. Continue the boot process; this command will force a file system repair and allow vCenter to load normally.