Apply changes on Harbor tile fails with "Error: Action Failed get_task: Task 5c822ca2-xxxxxxxxxx result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront."

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• When running "Apply Changes" in Operations Manager (Ops Manager), Harbor tile  fails with the following error message: 

 

#> Updating deployment:
 Expected task '58476' to succeed but state is 'error'
Exit code 1
Task 58476 | 09:40:40 | L executing pre-start: harbor-app/ed1d446d-7b0e-4857-a670-b08a2cda58bf (0) (canary) (00:00:19)
                    L Error: Action Failed get_task: Task 5c822ca2-bacf-4bc1-5f82-db8f711b6bc3 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.
Task 58476 | 09:40:46 | Error: Action Failed get_task: Task 5c822ca2-bacf-4bc1-5f82-db8f711b6bc3 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.
Task 58476 Started  Tue Aug 29 09:40:21 UTC 2023
Task 58476 Finished Tue Aug 29 09:40:46 UTC 2023
Task 58476 Duration 00:00:25
Task 58476 error


===== 2023-08-29 09:40:46 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=10.68.96.5 --deployment=harbor-container-registry-09d28774917ffa47fe22 deploy --no-redact /var/tempest/workspaces/default/deployments/harbor-container-registry-09d28774917ffa47fe22.yml"; Duration: 35s; Exit Status: 1
Exited with 1.
Exited with 1.

• You will notice below error in pre-start.stderr.log for harbor-container-registry. 

 

{code:java}
harbor-app/ed1d446d-7b0e-4857-a670-b08a2cda58bf:/var/vcap/jobs/harbor/bin# openssl s_client -showcerts -servername api.np-pksapi.globetel.com  -connect api.np-pksapi.globetel.com:8443
gethostbyname failure
connect:errno=0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/var/vcap/packages/python/python2.7/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/var/vcap/packages/python/python2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/var/vcap/packages/python/python2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/var/vcap/packages/python/python2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) server certificate verification failed. CAfile: /var/vcap/jobs/harbor/config/uaa_ca.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.10.3

Cause

The Harbor instance is configured with UAA authentication, it requires connecting to the UAA server first in the pre-start script, but it failed to establish a TLS  connection with the UAA server because the "uaa_ca.crt" is wrong.

Because the UAA certificate is rotated in server side, the CA certificate in Harbor tile should be reconfigured with the new UAA CA cert.

Resolution

1. Copy the  UAA cert from /var/vcap/jobs/uaa/config/ 

bosh -d pivotal-container-service-xxxx ssh pivotal-container-service/xxxx
sudo su # switch root user
cat /var/vcap/jobs/uaa/config/uaa.crt

2. Remove the private key section from uaa.crt (if exists) and paste the uaa.crt as a CA under Harbor tile "certificate" section.

3. Then click "Apply Changes" on Harbor tile .
4. After successful "Apply changes" , verify that UAA user can now login to Harbor console.

Additional Information

Impact/Risks:
Pods will start failing due to ImagePullBackOff error because Harbor is unreachable. 
 

Events:
  Type     Reason   Age                        From                                                      Message
  ----     ------   ----                       ----                                                      -------
  Warning  Failed   5m12s (x307 over 75m)      kubelet, xxxxx.compute.internal  Error: ImagePullBackOff
  Normal   BackOff  <invalid> (x419 over 75m)  kubelet, xxxxx.compute.internal  Back-off pulling image "harbor-xxxxx"