Error - Harbor Supervisor Service Inaccessible with 401 Unauthorized or I/O Timeout Errors after Reinstallation/Upgrade.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After a reinstallation or upgrade of the Harbor service in a vSphere Kubernetes Service (VKS) environment, the Harbor UI becomes inaccessible.

The core services fail to establish internal communication, resulting in the following symptoms:

• harbor-jobservice pods enter a CrashLoopBackOff or remain in Pending.
  
  
• Logs (Jobservice): Failed on load rest config err:http error: code 401, message {"errors": [{"code: "UNAUTHORIZED"}]}.
  
  
• Logs (Core): failed to connect to tcp://harbor-database:5432 ... i/o timeout.
  
  
• The Harbor package installation (pkgi) fails to reconcile.

Environment

vCenter Server 8.0 Update 3g

Supervisor v1.30.10+vmware.1

VKS 3.5.1+v1.34

Harbor Supervisor Service 2.13.1

Cause

The issue is typically caused by two factors following an upgrade or failed reinstallation:

• Credential Desynchronization - The password stored in the harbor-database (Postgres) does not match the secrets used by the harbor-core and harbor-jobservice pods.
  
  
• Configuration Gaps - The harbor-data-values YAML is missing mandatory network policies or TLS labels (managed-by: vmware-vRegistry) required for the Supervisor Service to properly route traffic between internal components.

Resolution

1. Synchronize the Database Password

You must manually update the Postgres password to match the secret expected by the application pods.

This command will make changes to your system. Review it carefully before running.

# Access the database container

kubectl exec -it -n svc-harbor-domain-c###### harbor-database-0 -c database -- /bin/bash

# Run the Postgres CLI to update the password# Note: Ensure the password matches the 'database-password' field in the harbor-database secret

psql -U postgres -c "ALTER USER postgres WITH PASSWORD '<required_password>';"

-- where <required_password> is the password you wish to set.

2. Update Configuration (Data-Values YAML)

Modify the Harbor configuration to include missing network and security metadata.

This command will make changes to your system. Review it carefully before running.

# Add to the 'network' or global section:

...
createNetworkPolicy: true
network:
  ipFamilies: ["IPv4"]

# Add under the 'tls' section (ensure 2-space indentation above ca.crt):

...
  tlsSecretLabels: {"managed-by": "vmware-vRegistry"}
...

3. Reset Trivy StatefulSet (If Applicable)

If the harbor-trivy pod is stuck due to a volume size change (e.g., from 20Gi to 50Gi), the StatefulSet must be recreated.

This command cannot be undone. Verify every parameter before running.

kubectl delete statefulset --cascade=orphan harbor-trivy -n svc-harbor-domain-c######

 

 

4. Force Package Reconciliation

Trigger the Supervisor Service to apply the new configurations.

This command will make changes to your system. Review it carefully before running.

# Pause the package reconciliation

kubectl edit pkgi harbor.tanzu.vmware.com -n vmware-system-supervisor-services

# Set 'pause: true', save and exit.

# Resume the reconciliation

kubectl edit pkgi harbor.tanzu.vmware.com -n vmware-system-supervisor-services

# Set 'pause: false', save and exit.

 

Additional Information

• Validation - After the reconciliation completes, check the pod status using 'kubectl get pods -n svc-harbor-domain-c######'.
  
  
• Admin Access - If you can reach the login page but cannot log in, restart the harbor-core deployment to ensure it has refreshed its connection pool with the updated database credentials.