Harbor pods crashing after upgrade

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Upgrading from Harbor version 2.09 to 2.13.1 causes database password issues. Pods show errors like below. 

 k get logs harbor-core-XXXX

[INFO] [/common/dao/base.go:67]: Registering database: type-PostgreSQL host-harbor-database port-5432 database-registry sslmode-"disable"
[ORM] Register db Ping `default`, failed to connect to `host=harbor-database user=postgres database=registry`: server error (FATAL: password authentication failed for user "postgres" (SQLSTATE 28P01))
[FATAL] [/core/main.go:190]: failed to initialize database: Register db Ping `default`, failed to connect to `host=harbor-database user=postgres database=registry`: server error (FATAL: password authentication failed for user "postgres" (SQLSTATE 28P01))

## k logs harbor-database-0

[132450] DETAIL:  unsupported
        Connection matched pg_hba.conf line 99: "host all all all md5"
 [132473] FATAL:  password authentication failed for user "postgres"
 [132473] DETAIL:  unsupported
        Connection matched pg_hba.conf line 99: "host all all all md5"

Environment

vSphere Supervisor 

VKS Standard Packages

Cause

Password mismatch between secrets and database. 

Resolution

Steps to resolve are to first confirm the password in the secret, like below. 

kubectl get secret -n tanzu-system-registry harbor-core-ver-1 -o jsonpath="{.data.POSTGRESQL_PASSWORD}" | base64 -d
<password>

You will then need to exec in to the database pod on update the database password from there. 

kubectl exec -it harbor-database-0 -n tanzu-system-registry -c database -- /bin/bash
env | grep POST
psql -U postgres
ALTER USER postgres WITH PASSWORD '<password from output>';

Restart database pod

kubectl delete pod n tanzu-system-registry harbor-database-0

Harbor core pods should then come back and Harbor UI is accessible. 

Additional Information

Harbor upgrades allow for skip of n-2 versions as per 
VKS Standard Packages Release Notes

Upgrade Path

    2.12.4+vmware.1-tkg.1 → 2.13.1+vmware.1-vks.1
    2.11.2+vmware.1-tkg.2 → 2.13.1+vmware.1-vks.1