Supervisor upgrade got stuck from v1.22.6 to v1.23.12

Products

• VMware vSphere with Tanzu

Issue/Introduction

Symptoms:

 

1. After Supervisor upgrade to v1.23.12 vCenter GUI shows still old version: v1.22.6 for Supervisor under "Current Version"

 

2. "kubectl get nodes" show that nodes were upgraded to the correct version

 

3.  Harbor deployment not running completely - DB Pod with Err Image Pull

 

kubectl get pods -A|grep -i harbor

vmware-system-registry-1223973458           harbor-1223973458-harbor-core-84975bf644-8s5cf                    0/1   CrashLoopBackOff  25 (116s ago)    140m
vmware-system-registry-1223973458           harbor-1223973458-harbor-database-0                               0/1     ErrImagePull       0                139m
vmware-system-registry-1223973458           harbor-1223973458-harbor-jobservice-7d86964f86-87qqm              1/1     Running     108 (5d12h ago)   5d22h
vmware-system-registry-1223973458           harbor-1223973458-harbor-nginx-7bcbbcb5d4-vrmz8                   1/1     Running     0                 5d22h
vmware-system-registry-1223973458           harbor-1223973458-harbor-portal-59c66cd58d-g92t5                  1/1     Running     0                 5d22h
vmware-system-registry-1223973458           harbor-1223973458-harbor-redis-0                                  1/1     Running     0                 5d22h
vmware-system-registry-1223973458           harbor-1223973458-harbor-registry-5988cc5878-k47vr                2/2     Running     0                 5d22h

 

 

 

 

 

Environment

vCenter 7.0.3 

Cause

harbor-database pod in CrashLoopBackOff - failed to pull images

Resolution

1. Delete harbor DB pod and once it reaches a running state restart wcp service

 

Example:

kubectl delete po harbor-1223973458-harbor-database-0 -n vmware-system-registry-1223973458"

 

Restart wcp service

ssh to vCenter and run command: service-control --restart wcp

 

2. Upgrade Supervisor again from VC GUI