Harbor Image Pull Fails with x509 Unknown Authority Due to Encrypted TLS Key in vSphere Kubernetes Service

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Production pods are unable to download images from Harbor. The following errors are observed during image pull operations:

  • Failed to pull image "...": x509: certificate signed by unknown authority

  • x509: invalid signature: parent certificate cannot sign this kind of certificate

• Additionally, the Harbor registry rollout may exhibit a "Pending" or "Unreconciled" state.

Environment

 

• VMware vSphere Kubernetes Service (VKS)

• Harbor 2.9.1

 

Cause

The custom TLS private key (tls.key) defined in the Harbor values.yaml configuration is encrypted. Harbor is unable to decrypt the key, failing to load the intended corporate certificate. Consequently, the system falls back to an auto-generated, self-signed certificate via cert-manager. Kubernetes worker nodes do not inherently trust this self-signed certificate, resulting in the x509 image pull failures.

Resolution

To resolve this issue, configure Harbor to use a valid, unencrypted certificate chain or manually establish trust for the self-signed certificate.

Option 1: Deploy a Decrypted Corporate Certificate (Recommended)

1. Acquire the password required to decrypt the existing private key, or generate a new, unencrypted tls.key.

2. Update the Harbor values.yaml configuration with the unencrypted private key and corresponding certificate.

3. Re-apply the Harbor configuration.

Option 2: Utilize the Auto-Generated Self-Signed Certificate

1. Remove the custom tlsCertificate configuration block from the Harbor values.yaml file to prevent the system from attempting to load the encrypted key.

2. Re-reconcile the Harbor packages to apply the updated configuration.

3. Manually configure all Kubernetes worker nodes within the cluster to explicitly trust the newly generated self-signed Certificate Authority (CA).

Verification Steps:

1. Confirm that all Harbor pods transition to a Running state.

2. Deploy a test pod and verify it successfully pulls an image from the Harbor registry without encountering x509 errors.