Harbor job is failing due to logrotate.conf and rsyslog_docker.conf​​​​​​​ created as directory instead of files

Products

• Pivotal CloudFoundry Services 1.x

Issue/Introduction

Symptoms:

·         Harbor job is failing when you are trying to recreate the vm using Bosh cck.

·         Bosh task log for vm recreation shows entries similar to:
Updating instance harbor-app: harbor-app/2df82ae5-22ee-4404-84cd-8704f9218213 (0) (canary) (00:00:55)
L Error: Action Failed get_task: Task c899ec83-1f0f-46d7-402e-d9c136938428 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

·         In the /var/vcap/sys/log/harbor/harbor-app-logs/pre-start.stderr.log file on the Harbor vm, you see the entries similar to:
Traceback (most recent call last):
File "main.py", line 64, in <module>
main()
File "main.py", line 35, in main prepare_log_configs(config_dict)
File "/usr/src/app/utils/log.py", line 26, in prepare_log_configs **config_dict)
File "/usr/src/app/utils/jinja.py", line 8, in render_jinja with open(dest, 'w') as f:IsADirectoryError: [Errno 21] Is a directory: '/config/log/logrotate.conf'

Environment

VMware PKS 1.x

Cause

When you use Bosh cck to recreate the vm when the Harbor instance is down, it creates the logrotate.conf & rsyslog_docker.conf as a directory instead of file. This causes the Harbor job to fail.

These folders/files exist under the directory /var/vcap/packages/harbor-app/common/config/log in the Harbor vm.

harbor-app: /var/vcap/packages/harbor-app/common/config/log# ls -al
drwxr-xr-x 2 root root 4096 Feb 27 01:46 logrotate.conf
drwxr-xr-x 2 root root 4096 Feb 27 01:46 rsyslog_docker.conf

Resolution

This is a known issue affecting Harbor 1.9.x and it will be fixed in future versions.

 

Workaround:

To work around this issue, delete both the folders causing this issue and apply changes from Ops Manager by selecting the Harbor.

harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f logrotate.conf
harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f rsyslog_docker.conf

Additional Information

In case above method doesn't work you can try below procedure:

 -- From the Harbor VM run below commands as 'root'

# alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock'
# alias docker-compose='/var/vcap/packages/docker-compose/bin/docker-compose -H unix:///var/vcap/sys/run/docker/dockerd.sock -f /var/vcap/packages/harbor-app/docker-compose.yml '

# docker stop $(docker ps -aq)
# docker rm $(docker ps -aq)

# docker-compose up -d
# docker-compose down -v
# cd /var/vcap/jobs/harbor/bin
# bash -x ./pre-start

-- Once above commands have run successfully, run 'Apply Changes' from the Opsman with only Bosh directed & Harbor ticked.