Harbor VM Bosh agents showing unavailable and restarting harbor service stalls then fails

Products

• VMware Tanzu Application Service

Issue/Introduction

Symptoms:

• Harbor is deployed via a Tile in a Tanzu Kubernetes Grid Integrated Edition (formerly VMware Enterprise PKS) installation.
• After preforming a monit stop on a bosh deployed Harbor VM the agents will not come back up.
• Redeploying the bosh VM with bosh cck or bosh deploy brings the Harbor VM back into the bad state
• You see messages similar to the following on the Harbor bosh VM in the /var/vcap/sys/log/harbor/ctl.stdout.log file.:

ERROR: for the harbor-db Cannot create container for service postgressql: Conflict. The Container name "/harbor-db" is already in use by container "4357ea1f8b67d467d4ea3d3s324dk49503k23fe2" . you have to remove (or rename) that container to be able to reuse that name

Cause

The Harbor tile uses the monit stop harbor command to stop Harbor, and it calls docker-compose down -v to stop the Harbor containers. Sometimes the docker containers will not stop properly and end up in an exited state instead of being removed. This causes the docker-compose up -v command when monit start is executed to fail indefinitely.

Resolution

1. Login to the Harbor tile VM using the bosh -d <Service_Instance_ID> ssh <Harbor-app_ID> command 

2. Run sudo -i to get to root mode privileges.

3. Stop the Harbor service by running the following command:
   
   monit stop harbor

NOTE: This will also stop the wavefront containers if it is enabled. Remember to run monit start wavefront when you start harbor at the end of this article

4. Issue the following commands to validate that there is a container stuck in an exiting or running state:
   
   alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock'
   
   docker ps -a

5. If there are leftover containers seen  in the docker ps -a output, preform the following step (if not proceed with step six):
   
   docker stop $(docker ps -aq) && docker rm $(docker ps -aq)

6. You can start the Harbor service without issues. Also start wavefront if it is enabled on harbor.
    

monit start harbor​​​​​​

monit start wavefront