ASM OPMS - /var full, made disk space available but services do not start
search cancel

ASM OPMS - /var full, made disk space available but services do not start


Article ID: 229207


Updated On:


CA App Synthetic Monitor


We found /var disk space fully utilized,  we deleted old content from the /var/lib/Docker

but the services do not start

opms logs report below error 

MISCONF Errors writing to the AOF file: No space left on device




the problem is caused by the brute-force deleting of /var/lib/docker:

org.openqa.selenium.WebDriverException: create container: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/668602ce0e276d58a3eb54ebd4e78dbf74fcdd596aa1e75237b72fabe8888277-init/merged: no such file or directory (WARNING: The server did not provide any stacktrace information)


1) stop all services including docker

monit stop all

check manually that nothing is really running : monit start all

systemctl stop docker

2) remove again that docker aof file

rm /var/lib/redis/appendonly.aof

3) delete directory /var/lib/docker

4) start docker and check it is running and not complaining

systemctl start docker

4) start all services

monit start redis

monit start all

Docker needs to rebuild the deleted structures. Check that errors in /var/log/smartpop/webdriver.log.0:

The docker services will report failures at the beginning because they'll start by downloading the docker images first.  It can take 30 minutes, it depends on the internet connectivity. Once all docker images are in place the services will be reported as green again. Do not restart the services often if the images are not in place, simply wait for the docker pull to complete. You can check the status by command:

docker images

And wait until all are in place again:

REPOSITORY                                                   TAG           IMAGE ID       CREATED                    jre8          f7d5abe922d6   15 months ago              83            278d9dfef4f4   16 months ago           72.0-10.2.2   c39e842e16e9   2 years ago     978MBaerokube/selenoid                                            1.9.2         dc39d7db4f7b   2 years ago   4.0           be0a197222e1   2 years ago          65.0-10.2.0   90b708f0bd3e   2 years ago             prod          644ba150ce85   5 years ago     509MB