Harbor upgrade from 1.8.x to 1.9.3 fails with harbor job
search cancel

Harbor upgrade from 1.8.x to 1.9.3 fails with harbor job

book

Article ID: 329107

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

  • Harbor upgrade from 1.8.x to 1.9.3 fails with an error similar to:
    Action Failed get_task: Task 1b4c82f6-cab2-41a9-46ae-d540e9e55a2e result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

  • You are unable to login to Harbor UI.

  • In the core.log file on harbor vm, you see the entries similar to:
    135 Nov 19 12:05:58 172.80.1.1 core[5878]: 2019-11-19T12:05:58Z [INFO] [/core/main.go:107]: Start to sync quota data .....                                                  
    136 Nov 19 12:05:58 172.80.1.1 core[5878]: 2019-11-19T12:05:58Z [INFO] /core/api/quota/migrator.go:90]: [Quota-Sync]:: start to ping server ... [chart]                   
    137 Nov 19 12:06:16 172.80.1.1 core[5878]: 2019-11-19T12:06:16Z [INFO] [/core/api/quota/migrator.go:92]: [Quota-Sync]:: fail to ping server ... [chart], quit sync ...     
    138 Nov 19 12:06:16 172.80.1.1 core[5878]: 2019-11-19T12:06:16Z [ERROR] [/core/main.go:109]: Fail to sync quota data, received unexpected status code: 403 <HTML><HEAD>

  • When you check the proxy settings in the Harbor tile configuration, you see below info listed:
    127.0.0.1,localhost,core,registry



Environment

VMware PKS 1.x

Cause

Before Harbor 1.9.0, the http_proxy/https_proxy setting is used in clair component to update the CVE data from internet. And the no_proxy setting was mentioned as:
127.0.0.1,localhost,core,registry

From Harbor 1.9.3, the http_proxy/https_proxy setting is populated to core, jobservice, clair components as well. And the no_proxy setting should contain all containers as below. Otherwise the core component can not communicate with the chartmuseum component.
127.0.0.1,localhost,.local,.internal,log,db,redis,nginx,core,portal,postgresql,jobservice,registry,registryctl,clair,chartmuseum,notary-server

When you try to upgrade Harbor tile which has the old no_proxy setting values, core components fails to access the chartmuseum component via proxy and causes upgrade failure.

Resolution

This is a known issue while upgrading Harbor from 1.8.x to 1.9.3.


Workaround:

To work around this issue, update the below info at Harbor tile > Networking > No Proxy field and then apply changes from Ops Manager.
127.0.0.1,localhost,.local,.internal,log,db,redis,nginx,core,portal,postgresql,jobservice,registry,registryctl,clair,notary-server,chartmuseum


Powered by Wolken Software