Configstore ramdisk is full' alerts on ESX host.
search cancel

Configstore ramdisk is full' alerts on ESX host.

book

Article ID: 312114

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
Due to a bug in ConfigStore API, stale data related to block devices might not be deleted in time from the ESXi ConfigStore database and causing an out of space condition. As a result, write operations to ConfigStore start to fail. In the backtrace, you see logs such as:

2022-12-19T03:51:42.733Z cpu53:26745174)WARNING: VisorFSRam: 203: Cannot extend visorfs file /etc/vmware/configstore/current-store-1-journal because its ramdisk (configstore) is full.

Environment

VMware vSphere ESXi 8.0.1
VMware vSphere ESXi 8.0.0
VMware vSphere ESXi 7.0.3

Cause

ConfigStore SetVitalDataInstances API sets vital configuration by overwriting existing data, sometimes this leads to empty rows in configstore database (stale data).

Resolution

Fix has been submitted to 7.0.3 P08, 8.0.1 P02, 8.0.2.

Workaround:

Follow the below mentioned steps.
Option 1:
Use configstore-recovery python script attached to the KB article. Copy the script to the host and run python configstore-recovery

The script performs following steps:
1. Temporarily increase the configstore ramdisk size to 64MB (initial size is 32MB)
2. Clean stale/empty data from the configstore DB.
3. Perform VACUUM on the configstore DB. The VACUUM command rebuilds the database file, repacking it into a minimal amount of disk space.
4. Revert configstore ramdisk size to 32MB

Logs from the script are captured in /var/run/log/syslog.log


Option 2:
Manually recover the host by following the below steps:

1. Temporarily increase the configstore ramdisk size to 64MB (initial size is 32MB).
a. Get configstore ramdisk group ID using:

vsish -e set /sched/groupPathNameToID host system visorfs ramdisks configstore

b. Set configstore ramdisk max memory to 64 using:

vsish -e set /sched/groups/<GID>/memAllocationInMB max=64

c. Verify configstore ramdisk memory allocation using:

vsish -e get /sched/groups/<GID>/memAllocationInMB

Example:
[root@sc2-10-186-8-178:~] vsish -e set /sched/groupPathNameToID host system visorfs ramdisks configstore
1627
[root@sc2-10-186-8-178:~]
[root@sc2-10-186-8-178:~] vsish -e get /sched/groups/1627/memAllocationInMB
memsched-allocation {
min:32
max:32
shares:-3
minLimit:-1
units: 4 -> mb
}
[root@sc2-10-186-8-178:~] vsish -e set /sched/groups/1627/memAllocationInMB max=64
[root@sc2-10-186-8-178:~]
[root@sc2-10-186-8-178:~] vsish -e get /sched/groups/1627/memAllocationInMB
memsched-allocation {
min:32
max:64
shares:-3
minLimit:-1
units: 4 -> mb
}
[root@sc2-10-186-8-178:~]


2. Forcefully purge any stale device entries currently on the host. This gives a chance to purge any recently unmapped devices (< 7days) to get purged

esxcli storage core device purge -f

3. Delete 'esx/storage/devices_access' configuration using configstorecli

configstorecli config current delete -c esx -g storage -k devices_access --all

4. Reboot the host (do not force reboot).

reboot