Supervisor upgrade got stuck from v1.22.6 to v1.23.12

Products

• VMware vSphere with Tanzu

Issue/Introduction

1. After Supervisor upgrade to v1.23.12 vCenter GUI shows still old version: v1.22.6 for Supervisor under "Current Version"
2. "kubectl get nodes" shows that nodes were upgraded to the correct version
3. Harbor deployment not running completely:
   
   

   # kubectl get pods -A | grep -i harbor
   vmware-system-registry-1223973458    harbor-1223973458-harbor-core-84975bf644-8s5cf          0/1   CrashLoopBackOff   25 (116s ago)    140m
   vmware-system-registry-1223973458    harbor-1223973458-harbor-database-0                     0/1   ErrImagePull       0                139m
   vmware-system-registry-1223973458    harbor-1223973458-harbor-jobservice-7d86964f86-87qqm    1/1   Running            108 (5d12h ago)  5d22h
   vmware-system-registry-1223973458    harbor-1223973458-harbor-nginx-7bcbbcb5d4-vrmz8         1/1   Running            0                5d22h
   vmware-system-registry-1223973458    harbor-1223973458-harbor-portal-59c66cd58d-g92t5        1/1   Running            0                5d22h
   vmware-system-registry-1223973458    harbor-1223973458-harbor-redis-0                        1/1   Running            0                5d22h
   vmware-system-registry-1223973458    harbor-1223973458-harbor-registry-5988cc5878-k47vr      2/2   Running            0                5d22h 

Environment

VMware vCenter Server 7.0.x 

Cause

Harbor deployment failed to be running due to ErrImagePull for database pod.

Resolution

1. Delete harbor database pod:
   
   Example:
   kubectl delete po harbor-1223973458-harbor-database-0 -n vmware-system-registry-1223973458
   
   
2. SSH vCenter Server and restart wcp service
   
   # service-control --restart wcp
   
   
3. Upgrade Supervisor again from vCenter Server GUI