Harbor job is failing due to various conf files created as directory instead of files

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Harbor job is failing when you are trying to recreate the vm using Bosh cck.

· Bosh task log for vm recreation shows entries similar to:

Updating instance harbor-app: harbor-app/xxxxxx (0) (canary) (00:00:55)
L Error: Action Failed get_task: Task c899ec83-1f0f-46d7-402e-d9c136938428 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

·In the /var/vcap/sys/log/harbor/harbor-app-logs/pre-start.stderr.log file on the Harbor vm, you see the entries similar to:

Traceback (most recent call last):
File "main.py", line 64, in <module>
main()
File "main.py", line 35, in main prepare_log_configs(config_dict)
File "/usr/src/app/utils/log.py", line 26, in prepare_log_configs **config_dict)
File "/usr/src/app/utils/jinja.py", line 8, in render_jinja with open(dest, 'w') as f:IsADirectoryError: [Errno 21] Is a directory: '/config/log/logrotate.conf'

Or

Traceback (most recent call last):
  File "/usr/src/app/main.py", line 15, in <module>
    cli()
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/commands/prepare.py", line 41, in prepare
    prepare_portal(config_dict)
  File "/usr/src/app/utils/portal.py", line 13, in prepare_portal
    render_jinja(
  File "/usr/src/app/utils/jinja.py", line 16, in render_jinja
    with open(dest, 'w') as f:
         ^^^^^^^^^^^^^^^
IsADirectoryError: [Errno 21] Is a directory: '/config/portal/nginx.conf'

Environment

VMware Harbor Registry tile

Cause

When you use Bosh cck to recreate the vm when the Harbor instance is down, it creates for example logrotate.conf & rsyslog_docker.conf & nginx.conf as a directory instead of file, This causes the Harbor job to fail.

These folders/files exist under the directory /var/vcap/packages/harbor-app/common/config/ in the Harbor vm. In the example below the files are actual folders

harbor-app: /var/vcap/packages/harbor-app/common/config/log# ls -al
drwxr-xr-x 2 root root 4096 Feb 27 01:46 logrotate.conf
drwxr-xr-x 2 root root 4096 Feb 27 01:46 rsyslog_docker.conf

harbor-app: /var/vcap/packages/harbor-app/common/config/portal# ls -al
drwxr-xr-x 2 root root 4096 Feb 27 01:46 nginx.conf

Determine in each of the subfolders in /var/vcap/packages/harbor-app/common/config/ if there are conf files that are actually folders below example shows number of .conf files present as a folder.

find . -type d -name "*"
.
./log
./core
./core/certificates
./db
./shared
./shared/trust-certificates
./registry
./nginx
./nginx/conf.d
./trivy-adapter
./registryctl
./jobservice
./portal

Resolution

There is no permanent fix on this problem as this might occur due to various reasons like frozen docker daemon leaving some artefacts in a bad state.

Workaround:

To work around this issue, delete the folders causing this issue and apply changes from Ops Manager by selecting the Harbor Tile.

harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f logrotate.conf
harbor-app: /var/vcap/packages/harbor-app/common/config/log# rm -f rsyslog_docker.conf
harbor-app: /var/vcap/packages/harbor-app/common/config/portal# rm -f nginx.conf

Additional Information

In case above method doesn't work you can try below procedure:

-- From the Harbor VM run below commands as 'root'

# alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock'
# alias docker-compose='/var/vcap/packages/docker-compose/bin/docker-compose -H unix:///var/vcap/sys/run/docker/dockerd.sock -f /var/vcap/packages/harbor-app/docker-compose.yml '

# docker stop $(docker ps -aq)
# docker rm $(docker ps -aq)

# docker-compose up -d
# docker-compose down -v
# cd /var/vcap/jobs/harbor/bin
# bash -x ./pre-start

In case the pre-start script fails review the error and in case you see other files with same message apply the process from above.

Once above commands have run successfully, run 'Apply Changes' from the Ops Manager with only Bosh Director & Harbor ticked.