File system /storage/log is not writeable alert in vCenter Server Appliance Management Interface (VAMI)

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter Server Appliance Management Interface (VAMI) displays a critical health alert indicating that the /storage/log file system is not writeable.
  
  
  
  
• When the status of the services is checked in an SSH session to the vCenter, the following error is observed:

# service-control --status --all

Traceback(most recent call last):

  File "/usr/bin/service-control", line 192, in <module>

    setupLogging('service-control', logMechanism='file', logDir=logDir)

  File "/usr/lib/vmware/site-packages/cis/utils.py", line 149, in__init__

    self.setupFileLogging(program, level=level, logDir=logDir,

  File "/usr/lib/vmware/site-packages/cis/utils.py", line 184, in setupFileLogging

    fileHandler = logging.FileHandler(logFile)

  File "/usr/lib/python3.10/logging/__init__.py", line 1169, in__init__

    StreamHandler.__init__(self, self._open ())

  File "/usr/lib/python3.10/logging/__init__.py", line 1201, in  _open

     return open_func(self.baseFilename, self.mode,

OSError: [Errno 30] Read-only file system: '/var/log/vmware/cloudvm/service-control.log'

• vCenter Server Appliance console displays a similar error:
  
  

Environment

VMware vCenter Server 8.x

Cause

This behavior typically stems from one of the following factors:

• File System Corruption: vCenter system files may be corrupted or locked in a read-only state.

• Emergency Mode: The vCenter appliance may have booted into emergency mode.

• Storage Instability: Intermittent storage disruptions may be affecting the appliance.

• Improper Shutdown: The vCenter server may have experienced an unexpected reboot or sudden power loss.

Resolution

1. Take a snapshot of the vCenter VM. If configured in Enhanced Linked Mode, take offline snapshots of all linked vCenters.
   Refer: Creating a vCenter snapshot in Enhanced Linked Mode (ELM)
   
   
2. Reboot the vCenter appliance.
   
   
3. Interrupt the boot process by pressing the e key to enter the GRUB menu.
   
   
4. Append fsck.repair=yes to the end of the kernel boot line, as outlined in Broadcom KB "Failed to start File System Check on /dev/vg_root_0/lv_root0" error on Photon OS based virtual appliances.
   
   
5. Resume the boot process to force the file system repair and allow the appliance to load normally.