ASM OPMS - /var full, made disk space available but services do not start

Products

• CA App Synthetic Monitor

Issue/Introduction

We found /var disk space fully utilized,  we deleted old content from the /var/lib/Docker

but the services do not start

opms logs report below error 

MISCONF Errors writing to the AOF file: No space left on device

Environment

 APP SYNTHETIC MONITOR (ASM)

Cause

the problem is caused by the brute-force deleting of /var/lib/docker:

org.openqa.selenium.WebDriverException: create container: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/668602ce0e276d58a3eb54ebd4e78dbf74fcdd596aa1e75237b72fabe8888277-init/merged: no such file or directory (WARNING: The server did not provide any stacktrace information)

Resolution

1) stop all services including docker

monit stop all

check manually that nothing is really running : monit start all

systemctl stop docker

2) remove again that docker aof file

rm /var/lib/redis/appendonly.aof

3) delete directory /var/lib/docker

4) start docker and check it is running and not complaining

systemctl start docker

4) start all services

monit start redis

monit start all

Docker needs to rebuild the deleted structures. Check that errors in /var/log/smartpop/webdriver.log.0:

The docker services will report failures at the beginning because they'll start by downloading the docker images first.  It can take 30 minutes, it depends on the internet connectivity. Once all docker images are in place the services will be reported as green again. Do not restart the services often if the images are not in place, simply wait for the docker pull to complete. You can check the status by command:

docker images

And wait until all are in place again:

REPOSITORY                                                   TAG           IMAGE ID       CREATED         SIZEregistry.asm.saas.broadcom.com/asm/jmeter                    jre8          f7d5abe922d6   15 months ago   135MBregistry.asm.saas.broadcom.com/asm/fpm/chromium              83            278d9dfef4f4   16 months ago   691MBregistry.asm.saas.broadcom.com/asm/browsers/chrome           72.0-10.2.2   c39e842e16e9   2 years ago     978MBaerokube/selenoid                                            1.9.2         dc39d7db4f7b   2 years ago     16.8MBregistry.asm.saas.broadcom.com/asm/selenoid/video-recorder   4.0           be0a197222e1   2 years ago     30.4MBregistry.asm.saas.broadcom.com/asm/browsers/firefox          65.0-10.2.0   90b708f0bd3e   2 years ago     947MBregistry.asm.saas.broadcom.com/asm/smartpop-rbtm             prod          644ba150ce85   5 years ago     509MB