File system/storage/dblog is not writable
search cancel

File system/storage/dblog is not writable

book

Article ID: 412796

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • You receive critical health alert "File system /storage/dblog is not writable" in VAMI interface. 

  • Checking service status of the vCenter fails with below error.

    # service-control --status --all
    Traceback (most recent call last):
      File "/usr/bin/service-control", line 178, in <module>
        setupLogging('service-control', logMechanism='file', logDir=logDir)
      File "/usr/lib/vmware/site-packages/cis/utils.py", line 145, in __init__
        rotate_bytes=rotate_bytes, rotate_count=rotate_count)
      File "/usr/lib/vmware/site-packages/cis/utils.py", line 176, in _setupFileLogging
        fileHandler = logging.FileHandler(logFile)
      File "/usr/lib/python3.7/logging/__init__.py", line 1087, in __init__
        StreamHandler.__init__(self, self._open())
      File "/usr/lib/python3.7/logging/__init__.py", line 1116, in _open
        return open(self.baseFilename, self.mode, encoding=self.encoding)
    OSError: [Errno 30] Read-only file system: '/var/log/vmware/cloudvm/service-control.log'


  • Restarting services fails with same below error.

    # service-control --stop && service-control --start
    Traceback (most recent call last):
      File "/usr/bin/service-control", line 178, in <module>
        setupLogging('service-control', logMechanism='file', logDir=logDir)
      File "/usr/lib/vmware/site-packages/cis/utils.py", line 145, in __init__
        rotate_bytes=rotate_bytes, rotate_count=rotate_count)
      File "/usr/lib/vmware/site-packages/cis/utils.py", line 176, in _setupFileLogging
        fileHandler = logging.FileHandler(logFile)
      File "/usr/lib/python3.7/logging/__init__.py", line 1087, in __init__
        StreamHandler.__init__(self, self._open())
      File "/usr/lib/python3.7/logging/__init__.py", line 1116, in _open
        return open(self.baseFilename, self.mode, encoding=self.encoding)
    OSError: [Errno 30] Read-only file system: '/var/log/vmware/cloudvm/service-control.log'


  • Under /var/log/vmware/vmon/vmon.log, below log snippets are available.

    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <rbd> Skip service health check. State STOPPED, Curr request 0
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <pschealth> Running the API Health command as user pschealth
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <pschealth-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-pschealth/vmon/pschealth-vmon-apihealth.py
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX Successfully executed service batch operation API_HEALTH.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <analytics> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <applmgmt> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <certificateauthority> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <certificatemanagement> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <cis-license> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <content-library> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <eam> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <envoy> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <hvc> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <imagebuilder> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <infraprofile> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <lookupsvc> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <netdumper> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <observability-vapi> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <perfcharts> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <pschealth> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <rbd> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <rhttpproxy> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <sca> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <sps> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <statsmonitor> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <sts> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <topologysvc> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <trustmanagement> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <updatemgr> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vapi-endpoint> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vcha> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vlcm> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vmcam> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vmcam> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vmonapi> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vmware-postgres-archiver> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vmware-vpostgres> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vpxd> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vpxd-svcs> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vsan-health> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vsm> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vsphere-ui> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vstats> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <vtsdb> Reset fail counters of service.
    YYYY-MM-DDTHH:MM:SS In(05) host-XXXX <wcp> Reset fail counters of service.
    (END)

Cause

There are multiple causes for this issue such as,

  • The vCenter system files have become corrupted/read-only.
  • vCenter is already in an emergency mode.
  • Due to an intermittent storage issue.
  • Due to an unexpected reboot/shutdown of the vCenter server.

Resolution

Note: Take Snapshot of the vCenter server (Offline Snapshots of all the vCenter servers in case of linked mode).

Reboot the vCenter server.

Additional Information

If the vCenter servers enters into emergency mode after reboot, please follow "[FAILED] Failed to start File System Check on /dev/vg_root_0/lv_root0" error on Photon OS based virtual appliances to resolve the issue.

Powered by Wolken Software