Harbor job is failing due to various conf files created as directory instead of files

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Harbor job is failing when you are trying to recreate the vm using Bosh cck.

·         Bosh task log for vm recreation shows entries similar to:
Updating instance harbor-app: harbor-app/2df82ae5-22ee-4404-84cd-8704f9218213 (0) (canary) (00:00:55)
L Error: Action Failed get_task: Task c899ec83-1f0f-46d7-402e-d9c136938428 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

·         In the /var/vcap/sys/log/harbor/harbor-app-logs/pre-start.stderr.log file on the Harbor vm, you see the entries similar to:

Traceback (most recent call last):
File "main.py", line 64, in <module>
main()
File "main.py", line 35, in main prepare_log_configs(config_dict)
File "/usr/src/app/utils/log.py", line 26, in prepare_log_configs **config_dict)
File "/usr/src/app/utils/jinja.py", line 8, in render_jinja with open(dest, 'w') as f:IsADirectoryError: [Errno 21] Is a directory: '/config/log/logrotate.conf'

 

Environment

TKGi 1.2x

Harbor 

Cause

When you use Bosh cck to recreate the vm when the Harbor instance is down, it creates for example logrotate.conf & rsyslog_docker.conf as a directory instead of file,  This causes the Harbor job to fail.

These folders/files exist under the directory /var/vcap/packages/harbor-app/common/config/ in the Harbor vm. In the example below the log file are actual folders located in log folder

harbor-app: /var/vcap/packages/harbor-app/common/config/log# ls -al
drwxr-xr-x 2 root root 4096 Feb 27 01:46 logrotate.conf
drwxr-xr-x 2 root root 4096 Feb 27 01:46 rsyslog_docker.conf

Determine in each of the subfolders in /var/vcap/packages/harbor-app/common/config/ if there are conf files that are actually folders below example shows no .conf files present as a folder.

find . -type d -name "*"
.
./log
./core
./core/certificates
./db
./shared
./shared/trust-certificates
./registry
./nginx
./nginx/conf.d
./trivy-adapter
./registryctl
./jobservice
./portal

 

Resolution

There is no permanent fix on this problem as this might occur due to various reasons like frozen docker daemon leaving some artefacts in a bad state.  

Workaround:

To work around this issue, delete the folders causing this issue and apply changes from Ops Manager by selecting the Harbor.

harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f logrotate.conf
harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f rsyslog_docker.conf

Additional Information

In case above method doesn't work you can try below procedure:

 -- From the Harbor VM run below commands as 'root'

# alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock'
# alias docker-compose='/var/vcap/packages/docker-compose/bin/docker-compose -H unix:///var/vcap/sys/run/docker/dockerd.sock -f /var/vcap/packages/harbor-app/docker-compose.yml '

# docker stop $(docker ps -aq)
# docker rm $(docker ps -aq)

# docker-compose up -d
# docker-compose down -v
# cd /var/vcap/jobs/harbor/bin
# bash -x ./pre-start

In case the pre-start script fails review the error and in case you see other files with same message apply the process from above.

Once above commands have run successfully, run 'Apply Changes' from the Opsman with only Bosh director & Harbor ticked.