Following the TKG cluster upgrade, updates to packages like cert-manager, contour, and harbor. Unable to perform a docker login.
search cancel

Following the TKG cluster upgrade, updates to packages like cert-manager, contour, and harbor. Unable to perform a docker login.

book

Article ID: 345705

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid

Issue/Introduction

  • Cert-manager has partially issued certificates for the harbor package, resulting in connection issues.
  • From the harbor-core pod, error message, it appears that the issuer name does not match the subject from the issuing certificate.
  • Following the TKG cluster upgrade and updates to packages like cert-manager, contour, and harbor, you might be unable to perform a Docker login to the Harbor instance (that is deployed as a package).

Environment

VMware Tanzu Kubernetes Grid 1.x

Tanzu Package deployed Harbor

Cause

This issue arose due to a cert-manager API version change(v1alpha2 -> v1) introduced during the package upgrade. After the upgrade, cert-manager should have reissued all the certificates used by the harbor components for internal communication. However, only the cert for harbor-core was issued, leaving the remaining certificates unchanged and putting harbor in an incomplete state. Additionally, the certificate specification changed, deprecating the "organization" field in certificates generated by the newer version of cert-manager, leading to a subject mismatch.

For example:

 In TKG 1.4, Subject: O = Project Harbor, CN = harbor-core

 In TKG 1.5, Subject: CN = harbor-core

Resolution

This is a known issue affecting the Harbor package in TKG1.5.


Workaround:

For any upgrade case involving cert-manager and harbor, please double-check the data age of the certificate and secret for the harbor.

If we encounter a situation like what was observed below, where only the data age of harbor-core is changed:

# kubectl get secrets -n tanzu-system-registry

 NAME                           TYPE                                DATA AGE

 default-token-rhb6q             kubernetes.io/service-account-token 3    449d

 harbor-ca-key-pair              kubernetes.io/tls                   3    449d

 harbor-core-internal-tls        kubernetes.io/tls                   3     14h

 harbor-core-ver-1              Opaque                              6    449d

 harbor-database-ver-1          Opaque                              1    449d

 harbor-jobservice-internal-tls  kubernetes.io/tls                   3    449d

 harbor-jobservice-ver-1        Opaque                              2    449d

 harbor-notary-server-ver-1     Opaque                              2    449d

 harbor-notary-server-ver-2     Opaque                              2    449d

 harbor-notary-signer            kubernetes.io/tls                   3    449d

 harbor-portal-internal-tls      kubernetes.io/tls                   3    449d

 harbor-registry-htpasswd       Opaque                              1    20h

 harbor-registry-internal-tls    kubernetes.io/tls                   3    449d

 harbor-registry-ver-1          Opaque                              3    449d

 harbor-registry-ver-2          Opaque                              2    20h

 harbor-tls                      kubernetes.io/tls                   3    449d

 harbor-token-service            kubernetes.io/tls                   3    449d

 harbor-trivy-internal-tls       kubernetes.io/tls                   3    449d

 harbor-trivy-ver-1             Opaque                              2    449d

 

please follow these steps to resolve the problem.

1. Delete the following certificates

# kubectl delete certificate harbor-core-internal-cert harbor-jobservice-internal-cert harbor-notary-signer-cert harbor-portal-internal-cert harbor-registry-internal-cert harbor-token-service-cert harbor-trivy-internal-cert -n tanzu-system-registry

2. Delete the following secrets

# kubectl delete secret harbor-core-internal-tls harbor-jobservice-internal-tls harbor-notary-signer harbor-portal-internal-tls harbor-registry-internal-tls harbor-token-service harbor-trivy-internal-tls -n tanzu-system-registry

3. Monitor the reissuing of certificates

# kubectl get certificate -n tanzu-system-registry

4. Watch the output for more than 10 minutes to ensure that all certificates are reissued. 

5. Delete all pods in the tanzu-system-registry namespace

kubectl delete pods --all -n tanzu-system-registry.

6. Wait for more than 5 minutes for all pods to become ready.

7. Perform a Docker login/push and pull to verify that the issue is resolved.



Additional Information

Impact/Risks:

Unable to perform a Docker login causing issues pulling/pushing images to the Harbor registry.