container name harbor-db is already in use by container

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• Harbor is inaccessible due to /var/vcap/store filesystem reaching 100% utilization.
• Manual clean up has been executed and capacity utilization has dropped.

harbor-app/6c75a99a-93ce-4411-a8bb-1634c17b4bbb:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1        99G   70G   24G  75% /var/vcap/store

• Harbor deployment is failing as harbor job is not running, restarting harbor VM doesn't change the state.

harbor-app/6c75a99a-93ce-4411-a8bb-1634c17b4bbb failing TKGI-Mgmt       10.72.1.14      vm-9c90148d-36b0-4d4e-866e-04f201e6ab2a large.disk      true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.265

harbor-app/6c75a99a-93ce-4411-a8bb-1634c17b4bbb:~# monit summary
Process 'dockerd'                   running
Process 'harbor'                    not monitored - start pending
Process 'bosh-dns'                  running
Process 'bosh-dns-resolvconf'       running
Process 'bosh-dns-healthcheck'      running
Process 'system-metrics-agent'      running
Process 'harbor-enable-bosh-dns'    running
System 'system_52a71217-8da4-4920-86e5-0936af78cbe9' running

Cause

Harbor job fails to be reinitialized due to the below error:
/var/vcap/sys/log/harbor/ctl.stdout.log:

ERROR: for harbor-db  Cannot create container for service postgresql: Conflict. The container name "/harbor-db" is already in use by container "bb8689e49a4b1f84744d293781fc06c4288f5b2b05ab9675eb761d65de305fe2". You have to remove (or rename) that container to be able to reuse that name.
ERROR: for postgresql  Cannot create container for service postgresql: Conflict. The container name "/harbor-db" is already in use by container "bb8689e49a4b1f84744d293781fc06c4288f5b2b05ab9675eb761d65de305fe2". You have to remove (or rename) that container to be able to reuse that name.
Encountered errors while bringin

Check the running containers on the node:

harbor-app/6c75a99a-93ce-4411-a8bb-1634c17b4bbb:~# docker container ls -a
CONTAINER ID   IMAGE                                  COMMAND                  CREATED          STATUS                     PORTS                       NAMES
bb8689e49a4b   goharbor/harbor-db:v2.5.3              "/docker-entrypoint.…"   2 days ago       Exited (255) 4 hours ago                               harbor-db

harbor-db container has exited with an error "Exited (255)"

Resolution

Remove the errored container:

harbor-app/6c75a99a-93ce-4411-a8bb-1634c17b4bbb:~# docker rm bb8689e49a4b
bb8689e49a4b

Stop all jobs on the harbor VM:

monit stop all

Restart the deployment jobs: where DEPLOYMENT_NAME is the name of the Harbor deployment.

bosh -d DEPLOYMENT_NAME restart

Additional Information

Impact/Risks:
Harbor registry inaccessible.