Harbor Data Loss scenario: Upgrade of harbor-container-registry tile results in data loss. smoke-test errand shows error: "run-errand smoke-testing --when-changed" and "Exit Status: 1"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware Tanzu Kubernetes Grid Integrated Edition (Core)
• VMware Tanzu Kubernetes Grid Integrated Edition 1.x
• VMware Tanzu Kubernetes Grid Integrated EditionStarter Pack (Core)

Issue/Introduction

Scenario:

Data loss may occur if a Harbor tile is upgraded to an unsupported Harbor tile version or a version not within Harbor Supported Upgrade Paths.

 

Example:

• Your current Harbor tile version is v.2.7.0
  
  
• You wish to upgrade Harbor to v2.11.0
  
  
• The supported upgrade path is: v2.7.0 -> v2.9.x -> v2.11.0
  
  
• You upgrade Harbor directly from v2.7.0 to v2.11.0
  
  
• You Apply Changes via Opsmanager
  
  
• You Login to the Harbor web UI
  
  
• Your Harbor data is gone and it appears as if you have a brand new VMware Harbor Registry instance.

 

It is important to follow the documented Harbor upgrade path for your version.

 

To avoid this scenario:

• Refer to the VMware Harbor Registry Tile Release Notes
  
  
• Locate the section pertaining to your current Harbor version
  
  
• Reference Supported Upgrade Paths element in the table provided

 

NOTE: A Harbor improvement has already been submitted to the avoid these scenarios.

Improvement Details:

The harbor database should fail early when upgrading from an unsupported version

The plan is to utilize an attribute within the Harbor tile metadata.yml file. The attribute, minimum_version_for_upgrade, will be implemented to avoid this in future releases.

Environment

VMware Harbor Registry: versions v2.11.0 and earlier

Opsmanager: v3.0.32+LTS-T

Cause

Details:

Harbor upgrades may include upgrades to new Postgres database versions. Upgrades may also include Postgres database schema changes.

Any upgrade attempts which skip these changes may result in the Postgres database to become corrupt or lost.

 

Potential scenarios:

NOTE: There can be different scenarios depending on source and target Harbor versions.  Variables include, but not limited to:

• Postgres versions
  
  
• Harbor db schema differences 

 

Reported Scenario:

• Environment Details:
  
  
  • Source Harbor tile version is v2.7.0
    
    
  • Opsmanager version: v3.0.32+LTS-T
    
    
• Harbor tile is upgraded directly to v2.11.0
   
  • Supported Path v2.7.0 -> v2.9.x -> v2.11.0

• Data no longer exists in Harbor web UI

 

Scenario Details:

• In the above environment, the harbor-db cannot support upgrade from v2.7.0 to v2.11.0.
  
  
• Harbor v2.7.0 runs Postgres 13.
  
  
• Harbor v2.11.0 runs Postgres version 15 which only supports upgrade from Postgres 14. 
  
  NOTE: Postgres 14 is introduced in Harbor v2.9 and v2.10.

Resolution

Workaround:

There is a recovery procedure.  Recovery procedures may be different, depending upon the original Harbor version and the upgraded version.

Therefore, it is important to engage Tanzu Support.  To expedite, include the following in the support request:

 

• Login to your Opsmanager UI
  
  
  • Download the Opsmanager Support Bundle
    
    
  • From the Opsmanager UI
    
    
    • Locate the change log pertaining to the failed upgrade of VMware Harbor Registry
      
      • See: Change Log reference   
        
        
    • Copy the error information shown for Tanzu Support

• Open a Tanzu Support Request
  
  
• Mention this KB ID: 381135
  
  
• Attach all of the above to the Support Request

Additional Information

Product Enhancement: https://github.com/goharbor/harbor/issues/21099