Updating the Harbor certificate in Enterprise PKS fails with error "failed to apply Harbor config: Installation was unsuccessful. Error: Installing VMware Harbor Registry"

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

• You see the message "failed to apply Harbor config" when applying changes to the Harbor configuration.
• This issue occurs if this is the task is performed from the Enterprise PKS Management Console (EPMC) or Opsmanager.
• Harbor is using NFS-backed storage for harbor-registry.
• Once failed, the Harbor deployment VM is in a stopped state.
• monit summary shows no process running in the Harbor vm
• You see messages similar to the following from the bosh task command (run against the bosh task ID related to the Harbor configuration change):

Task 1854 | 22:04:51 | Updating instance harbor-app: harbor-app/2becda9c-b215-48f4-b932-7734f4c6a4b2 (0) (canary) (00:01:37)
L Error: Action Failed get_task: Task e0ce7d47-7741-4812-6499-319e7231ffca result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.
Task 1854 | 22:06:28 | Error: Action Failed get_task: Task e0ce7d47-7741-4812-6499-319e7231ffca result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

Task 1854 Started  Thu Feb 27 22:04:48 UTC 2020
Task 1854 Finished Thu Feb 27 22:06:28 UTC 2020
Task 1854 Duration 00:01:40
Task 1854 error

Capturing task '1854' output:
Expected task '1854' to succeed but state is 'error'

Exit code 1

• You see messages similar to the following in the /var/vcap/sys/log/harbor/pre-start.stderr.log file:

Removing network harbor-app_harbor-clair
Removing network harbor-app_harbor-notary
Removing network harbor-app_harbor-chartmuseum
Removing network harbor-app_notary-sig
Warning: Stopping rpcbind.service, but it can still be activated by:
rpcbind.socket
mount.nfs: /harbor-registry is busy or already mounted
Warning: Stopping rpcbind.service, but it can still be activated by:
rpcbind.socket
mount.nfs: /harbor-registry is busy or already mounted
(END)

• You see an error similar to the following if you try to manually start the Harbor VM via the bosh command:

Task 1871 | 22:24:18 | Updating instance harbor-app: harbor-app/2becda9c-b215-48f4-b932-7734f4c6a4b2 (0) (canary) (00:00:55)
L Error: Action Failed get_task: Task 041411e1-88ed-4890-6772-fe24064dc7f3 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.
Task 1871 | 22:25:13 | Error: Action Failed get_task: Task 041411e1-88ed-4890-6772-fe24064dc7f3 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

Task 1871 Started  Thu Feb 27 22:24:17 UTC 2020
Task 1871 Finished Thu Feb 27 22:25:13 UTC 2020
Task 1871 Duration 00:00:56
Task 1871 error

Changing state:
Expected task '1871' to succeed but state is 'error'

Exit code 1

Environment

VMware PKS 1.x

Cause

This issue can occur if the NFS share is already mounted and a re-mount attempt is made to the same folder. You can run the below command to confirm that the NFS share is already mounted. 

cat /proc/mounts | grep nfs

Note: You should see output similar to the following:

10.20.1.200:/vmware_harbor_repo /harbor-registry nfs4 rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.31.0.5,local_lock=none,addr=10.20.1.200 0 0

Resolution

This is a known issue affecting Harbor and there is currently no resolution.

Workaround:
To workaround the issue, manually unmount /harbor-registry by running the following commands:

sudo -i
umount /harbor-registry

Once this is done, apply the changes again from EPMC or Opsmanager.