VMware Harbor upgrade fails with an error "Failed to update database, error: pq: must be owner of relation schema_migrations"

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• When you initiate the VMware Harbor upgrade from Ops Manager in PKS environment, it fails with below error:

Failed to update database, error: pq: must be owner of relation schema_migrations

• In the Harbor instance, you see Notary container is not running.

• In the /var/vcap/sys/log/harbor-app-logs/notary-server.log file, you see the entries similar to:

Apr 11 09:12:25 172.30.1.1 notary-server[5973]: 2019/04/11 09:12:25 Failed to update database, error: pq: must be owner of relation schema_migrations
Apr 11 09:13:25 172.30.1.1 notary-server[5973]: 2019/04/11 09:13:25 Updating database.
Apr 11 09:13:25 172.30.1.1 notary-server[5973]: 2019/04/11 09:13:25 Failed to update database, error: pq: must be owner of relation schema_migrations
Apr 11 09:14:26 172.30.1.1 notary-server[5973]: 2019/04/11 09:14:26 Updating database.

Environment

VMware PKS 1.x

Cause

This issue occurs if the db owner for the notary-server and notary-signer does not match with the expected values, server & signer respectively.

Resolution

This is a known issue when you followed the Harbor upgrade path 1.5.x > 1.6.x> 1.7.x.

This issue will be resolved in VMware Harbor 1.7.5.

Workaround:

To work around this issue, updated the db owner info to default expected values:

1. Set Docker alias : alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock'

2. Run docker ps to see if notary-server and notary-signer containers are running or not.

3. Login to harbor-db container and connect to postgres by running:
   docker exec -it harbor-db /bin/bash
   psql -U postgres

4. Connect to notaryserver db and check the current owner:
   \c notaryserver
   \dt schema_migrations

5. Update the notaryserver db owner details as per below:
   ALTER TABLE tuf_files OWNER TO server;
   ALTER SEQUENCE tuf_files_id_seq OWNER TO server;
   ALTER TABLE change_category OWNER TO server;
   ALTER TABLE changefeed OWNER TO server;
   ALTER SEQUENCE changefeed_id_seq OWNER TO server;
   ALTER TABLE schema_migrations OWNER TO server;
   SELECT setval('changefeed_id_seq', max(id)) FROM changefeed;
   ALTER TABLE "changefeed" ALTER COLUMN "id" SET default nextval('changefeed_id_seq');

6. Now, connect to notarysigner db and check the current owner:
   \c notarysigner
   \dt schema_migrations

7. Update the notarysigner db owner details as per below:
   ALTER TABLE private_keys OWNER TO signer;
   ALTER SEQUENCE private_keys_id_seq OWNER TO signer;
   ALTER TABLE schema_migrations OWNER TO signer;

8. exit from the database and apply the changes from Ops Manager.
   \q