cert-manager partially issued certificates for the harbor package, resulting in connection issues.
From the harbor-core pod, error message, it appears that the issuer name does not match the subject from the issuing certificate.
Following the TKG cluster upgrade and updates to packages like cert-manager, contour, and harbor, you might be unable to perform a Docker login to the Harbor instance (that is deployed as a package)
The issue arose due to a cert-manager API version change(v1alpha2 -> v1) introduced during the package upgrade.
After the upgrade, cert-manager should have reissued all the certificates used by the harbor components for internal communication.
However, only the cert for harbor-core was issued, leaving the remaining certificates unchanged and putting harbor in an incomplete state.
Additionally, the certificate specification changed, deprecating the "organization" field in certificates generated by the newer version of cert-manager, leading to a subject mismatch.
For example:
In TKG 1.4, Subject: O = Project Harbor, CN = harbor-core
In TKG 1.5, Subject: CN = harbor-core
This is a known issue affecting the Harbor package in TKG1.5. Currently there is no resolution
For any upgrade case involving cert-manager and harbor, please double-check the data age of the certificate and secret for the harbor.
If we encounter a situation like what was observed below, where only the data age of harbor-core is changed:
# kubectl get secrets -n tanzu-system-registry
NAME TYPE DATA AGE
default-token-rhb6q kubernetes.io/service-account-token 3 449d
harbor-ca-key-pair kubernetes.io/tls 3 449d
harbor-core-internal-tls kubernetes.io/tls 3 14h
harbor-core-ver-1 Opaque 6 449d
harbor-database-ver-1 Opaque 1 449d
harbor-jobservice-internal-tls kubernetes.io/tls 3 449d
harbor-jobservice-ver-1 Opaque 2 449d
harbor-notary-server-ver-1 Opaque 2 449d
harbor-notary-server-ver-2 Opaque 2 449d
harbor-notary-signer kubernetes.io/tls 3 449d
harbor-portal-internal-tls kubernetes.io/tls 3 449d
harbor-registry-htpasswd Opaque 1 20h
harbor-registry-internal-tls kubernetes.io/tls 3 449d
harbor-registry-ver-1 Opaque 3 449d
harbor-registry-ver-2 Opaque 2 20h
harbor-tls kubernetes.io/tls 3 449d
harbor-token-service kubernetes.io/tls 3 449d
harbor-trivy-internal-tls kubernetes.io/tls 3 449d
harbor-trivy-ver-1 Opaque 2 449d
please follow these steps to resolve the problem.
# kubectl delete certificate harbor-core-internal-cert harbor-jobservice-internal-cert harbor-notary-signer-cert harbor-portal-internal-cert harbor-registry-internal-cert harbor-token-service-cert harbor-trivy-internal-cert -n tanzu-system-registry
# kubectl delete secret harbor-core-internal-tls harbor-jobservice-internal-tls harbor-notary-signer harbor-portal-internal-tls harbor-registry-internal-tls harbor-token-service harbor-trivy-internal-tls -n tanzu-system-registry
# kubectl get certificate -n tanzu-system-registry
Watch the output for more than 10 minutes to ensure that all certificates are reissued. *
kubectl delete pods --all -n tanzu-system-registry.
Wait for more than 5 minutes for all pods to become ready.
Unable to perform a Docker login where we cannot pull/push images to registry