Harbor Data Loss scenario: Upgrade of harbor-container-registry tile results in data loss. smoke-test errand shows error: "run-errand smoke-testing --when-changed" and "Exit Status: 1"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware Tanzu Kubernetes Grid Integrated Edition (Core)
• VMware Tanzu Kubernetes Grid Integrated Edition 1.x
• VMware Tanzu Kubernetes Grid Integrated EditionStarter Pack (Core)

Issue/Introduction

• Data loss may occur if a Harbor tile is upgraded to an unsupported Harbor tile version or a version not within Harbor Supported Upgrade Paths.
  
  
• Your current Harbor file version is v.2.7.0
  
  
• You wish to upgrade Harbor to v2.11.0
  
  
• The supported upgrade path is: v2.7.0 -> v2.9.x -> v2.11.0
  
  
• You upgrade Harbor directly from v2.7.0 to v2.11.0
  
  
• You applied changes via OpsManager
  
  
• Your Harbor data is gone, and it appears you have a brand-new VMware Harbor Registry instance.

Environment

Harbor Registry: 2.11.0 and earlier

Opsmanager: v3.0.32+LTS-T

Cause

Harbor upgrades may include upgrades to new Postgres database versions. Upgrades may also include changes to the Postgres database schema.

Any upgrade attempt that skips these changes may result in the Postgres database becoming corrupted or lost.

• Harbor-DB cannot support an upgrade from v2.7.0 to v2.11.0.
  
  
• Harbor v2.7.0 runs Postgres 13.
  
  
• Harbor v2.11.0 runs Postgres version 15, which only supports upgrade from Postgres 14.
  
  
• NOTE: Postgres 14 is introduced in Harbor v2.9 and v2.10.

Resolution

This procedure is only for Harbor v2.7.0 to v2.11.0 failure:

1. Note: the following commands are all executed in the vm of harbor-app.

2. Confirm original version was v2.7.0

3. Confirm current Harbor tile shows v2.11.0 (after upgrade) 

4. SSH to  the harbor-app

   bosh -d harbor-container-registry-UUID ssh harbor-app 

5. Change to root in the harbor-app vm

   sudo -i 

6. Stop Harbor App

   # Stop harbor
   monit stop harbor
   # Use summary to ensure harbor is stopped 
   monit summary

7. Initialize docker and docker-compose for harbor-app

   export PATH=$PATH:/var/vcap/jobs/harbor/packages/docker/bin:/var/vcap/jobs/harbor/packages/docker-compose/bin

   alias docker="docker -H unix:///var/vcap/sys/run/docker/dockerd.sock"
   
   alias docker-compose="docker-compose -H unix:///var/vcap/sys/run/docker/dockerd.sock -f /var/vcap/packages/harbor-app/docker-compose.yml"

8. Prepare the harbor-db:v2.10.0, which supports migrating from PostgreSQL 13 to 14

   docker pull goharbor/harbor-db:v2.10.0

   If we can't pull the goharbor/harbor-db:v2.10.0 image due to the network limitation, we can prepare the image using the following commands in the jumper vm

   docker pull goharbor/harbor-db:v2.10.0
   
   docker save goharbor/harbor-db:v2.10.0 -o harbor-db.tar
   
   bosh -d harbor-container-registry-7ddbfeb36ca038df9f79 scp harbor-db.tar harbor-app:/tmp/harbor-db.tar
   
   bosh -d harbor-container-registry-7ddbfeb36ca038df9f79 ssh harbor-app -c "sudo /var/vcap/jobs/harbor/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock load -i /tmp/harbor-db.tar"
   
   bosh -d harbor-container-registry-7ddbfeb36ca038df9f79 ssh harbor-app -c "rm /tmp/harbor-db.tar"

9. Edit docker-compose.yaml to use harbor-db:v2.10.0

   sed -i "s/harbor-db:v2.11.0/harbor-db:v2.10.0/g" /var/vcap/packages/harbor-app/docker-compose.yml

10. Remove PostgreSQL 15 data that doesn't have data for Harbor 2.7

    rm -rf /var/vcap/store/harbor/database/pg15

11. Ensure the /var/vcap/store/harbor/database/pg13 owner and group are runit-log.
    NOTE: This step is very Important

    ls -l /var/vcap/store/harbor/database/
    # Run the chown when owner:group if it is not runit-log:runit-log # chown -R runit-log:runit-log /var/vcap/store/harbor/database/pg13

12. Run harbor 2.11.0 with harbor-db:v2.10.0

    docker-compose up -d

13. Check the data of Harbor 2.7 has been recovered
    
    
14. Wait for the database to be migrated from 13 to 14

    # ls to ensure /var/vcap/store/harbor/database/pg13 is migrated to /var/vcap/store/harbor/database/pg14
    # migration is down when pg13 is gone
    ls /var/vcap/store/harbor/database/

15. We can also check the harbor-db logs to ensure they contain "upgrade DB from 13 to 14" and "Upgrade Complete."

    docker logs harbor-db

16. Log in to the Harbor ui to ensure the data of Harbor 2.7 is recovered
    
    
17. Migrate the database data from PostgreSQL 14 to 15
    
    

    docker-compose stop
    
    sed -i "s/harbor-db:v2.10.0/harbor-db:v2.11.0/g" /var/vcap/packages/harbor-app/docker-compose.yml
    
    docker-compose up -d

18. Check the harbor-db logs to ensure it contains "upgrade DB from 14 to 15."

    # ls to ensure /var/vcap/store/harbor/database/pg14 is migrated to /var/vcap/store/harbor/database/pg15
    ls /var/vcap/store/harbor/database/
    
    # check the logs of the harbor-db to ensure the it contains "upgrade DB from 14 to 15" and "Upgrade Complete"
    docker logs harbor-db

19. Log in to the harbor ui to ensure the data still exists
    
    
20. Start harbor-app

    docker-compose stop
    monit start harbor
    monit summary
    

 

 

Additional Information

Product Enhancement: 21099