Harbor process won't start with “container is marked for removal and cannot be started” error in Harbor 1.8

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

After clicking Apply Changes to update Harbor Tile configuration or upgrade to a new version, Operations Manager (Ops Manager) executes 'monit stop all' on Harbor VM. 'Monit stop harbor' calls 'docker-compose down' to stop Harbor.

However, before 'docker-compose down' exits completely, the 'docker-compose up' process created by the previous 'monit start harbor' already exits. Monit immediately detects the harbor job is stopped, and then stops the docker job.

Then 'docker-compose down' raises 'can not connect to docker deamon', and sometimes left harbor containers in dead status. The dead containers prevent Harbor service to start in the next 'monit start all'.

The following logs are recorded in /var/vcap/sys/log/harbor/ctl.stdout.log.

Starting harbor-log ... 

ERROR: for harbor-log  Cannot start service log: container is marked for removal and cannot be started

ERROR: for log  Cannot start service log: container is marked for removal and cannot be started
Encountered errors while bringing up the project.

Environment

Product Version: 1.8

Resolution

This is a known issue and appears in Harbor 1.8.

To work around this issue, remove the 'Dead' containers and use monit to restart harbor process.

Here are procedures to remove the 'Dead' containers.

1. Find the docker binary.

# find /var/vcap/data/packages -name docker
/var/vcap/data/packages/docker
/var/vcap/data/packages/docker/2ee5307091baf5e28d7ec7b80209cd4fac331aaf/bin/docker

2. Create an alias for convenience.

# alias docker=/var/vcap/data/packages/docker/2ee5307091baf5e28d7ec7b80209cd4fac331aaf/bin/docker

3. Setup the environment.

# export DOCKER_HOST=unix:///var/vcap/sys/run/docker/dockerd.sock

4. List the containers with the status 'Dead'.

# docker container ls -a

5. Remove these 'Dead' containers.

# docker container rm container_id

If there are no containers listed out in step 4, we can use the following command to clean up them.

# docker container prune