Harbor Registry is down: Harbor pods fail to create in tanzu-system-registry due to Pod Security Admission (PSA) violations and reverted namespace labels
search cancel

Harbor Registry is down: Harbor pods fail to create in tanzu-system-registry due to Pod Security Admission (PSA) violations and reverted namespace labels

book

Article ID: 424330

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

The Harbor Registry is reported as down, which may cause issues for the business and production environments. Users encounter an error where Harbor registry pods are failing to create in the tanzu-system-registry namespace.

Impacted users report following the techdoc Configure PSA for TKR 1.25 and Later to add labels to the namespace (to allow pod creation), but observe that labels are being removed after a short period of time and pods begin failing again. This behavior is typically seen on VKS clusters running version 1.26 and later.

Cause

There are two primary drivers for this behavior:

  1. Version Incompatibility: The installed version of Harbor (e.g., v2.6 or older) is incompatible with Kubernetes  (e.g., 1.32). Modern TKr versions enforce a "restricted" PSA mode by default which older Harbor manifests do not meet.
  2. Reconciliation Conflict: The kapp-controller manages the state of the Tanzu packages. When a user manually modifies namespace labels to bypass PSA, the controller identifies a drift from the defined PackageInstall configuration and automatically reverts the labels to the original state, causing the pods to fail again.

Resolution

To resolve this issue and restore the Harbor Registry, perform the following steps:

  1. Upgrade Tanzu CLI: Ensure you are using the latest version of the Tanzu CLI, as older versions may not support the necessary packages.
  2. Upgrade Harbor Package: Upgrade the Harbor package to a supported version. For TKr 1.32 compatibility, Harbor must be between version 2.9 and 2.14. These versions include the necessary security contexts to comply with restricted PSA standards natively.
  3. Workaround (PSA Configuration): If an immediate upgrade is not possible, you can configure the podSecurityStandard cluster-wide using the Cluster-v1beta1 API. This prevents the reconciliation conflict by defining the security standard at the cluster level rather than the namespace level.
  4. Verify Package Reconciliation: Ensure that Harbor, Contour, and Cert-Manager PackageInstalls are no longer showing "Reconcile failed" errors and that all pods and containers are Running.

Additional Information

References

Powered by Wolken Software