· Harbor job is failing when you are trying to recreate the vm using Bosh cck.
· Bosh task log for vm recreation shows entries similar to:
Updating instance harbor-app: harbor-app/2df82ae5-22ee-4404-84cd-8704f9218213 (0) (canary) (00:00:55)
L Error: Action Failed get_task: Task c899ec83-1f0f-46d7-402e-d9c136938428 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.
· In the /var/vcap/sys/log/harbor/harbor-app-logs/pre-start.stderr.log file on the Harbor vm, you see the entries similar to:
Traceback (most recent call last):
File "main.py", line 64, in <module>
main()
File "main.py", line 35, in main prepare_log_configs(config_dict)
File "/usr/src/app/utils/log.py", line 26, in prepare_log_configs **config_dict)
File "/usr/src/app/utils/jinja.py", line 8, in render_jinja with open(dest, 'w') as f:IsADirectoryError: [Errno 21] Is a directory: '/config/log/logrotate.conf'
When you use Bosh cck to recreate the vm when the Harbor instance is down, it creates the logrotate.conf & rsyslog_docker.conf as a directory instead of file. This causes the Harbor job to fail.
These folders/files exist under the directory /var/vcap/packages/harbor-app/common/config/log in the Harbor vm.
harbor-app: /var/vcap/packages/harbor-app/common/config/log# ls -al
drwxr-xr-x 2 root root 4096 Feb 27 01:46 logrotate.conf
drwxr-xr-x 2 root root 4096 Feb 27 01:46 rsyslog_docker.conf
This is a known issue affecting Harbor 1.9.x and it will be fixed in future versions.
To work around this issue, delete both the folders causing this issue and apply changes from Ops Manager by selecting the Harbor.
harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f logrotate.conf
harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f rsyslog_docker.conf
# alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock' # alias docker-compose='/var/vcap/packages/docker-compose/bin/docker-compose -H unix:///var/vcap/sys/run/docker/dockerd.sock -f /var/vcap/packages/harbor-app/docker-compose.yml ' # docker stop $(docker ps -aq) # docker rm $(docker ps -aq) # docker-compose up -d # docker-compose down -v # cd /var/vcap/jobs/harbor/bin # bash -x ./pre-start-- Once above commands have run successfully, run 'Apply Changes' from the Opsman with only Bosh directed & Harbor ticked.