Attempting to expand Harbor Database or Harbor-redis pods component size fails and package reconciliation status goes into "not reconciled" state
search cancel

Attempting to expand Harbor Database or Harbor-redis pods component size fails and package reconciliation status goes into "not reconciled" state

book

Article ID: 376220

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Attempting to expand Harbor Database or Redis pods fails due to the volumes being stateful sets.
In order to increase, this it needs to be completed in offline mode by scaling down the stateful sets and then completing the increase of the volumes. 

After attempting expansion, on running the tanzu package installed get command, you see the below error
 

tanzu package installed get harbor -n harbor
NAME: harbor
PACKAGE-NAME: harbor.tanzu.vmware.com
PACKAGE-VERSION: 2.7.1+vmware.1-tkg.1
STATUS: Reconcile failed: Error (see .status.usefulErrorMessage for details)
CONDITIONS: [
{ReconcileFailed True Error (see .status.usefulErrorMessage for details)}]
USEFUL-ERROR-MESSAGE:] Waited for 1.xs due to client-side throttling, not priority and fairness, request: GET:https://<ClusterIP-service>:443/apis/authentication.concierge.pinniped.tmc.cloud.vmware.com/v1alpha1
kapp: Error: update statefulset/harbor-redis (apps/v1) namespace: harbor:
Updating resource statefulset/harbor-redis (apps/v1) namespace: harbor:
API server says:
StatefulSet.apps "harbor-redis" is invalid: spec:
Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden (reason: Invalid)

Environment

vSphere CSI Driver

Cause

vSphere CSI driver does not support expansion of PV that is related to a pod created by Stateful Set either by online mode or offline mode.

Resolution

Resize of Stateful Set volume is still under development in Kubernetes and therefore not yet supported. However steps to complete can be found below. 


First we need to change harbor instance to read-only mode
Notice: this would cause downtime during this update
Login as admin => Navigate to Configuration => System Settings => Repository Read Only

Step 1 scale sts to 0
Depending on what you are increase you may need to increase for both database and redis

kubectl scale sts/harbor-database --replicas=0 -n tanzu-system-registry
kubectl scale sts/harbor-redis --replicas=0 -n tanzu-system-registry

Step 2 resize pvc, size set to 2GB

kubectl patch pvc database-data-harbor-database-0 --patch '{"spec": {"resources": {"requests": {"storage": "2Gi" }}}}' -n tanzu-system-registry
kubectl patch pvc data-harbor-redis-0 --patch '{"spec": {"resources": {"requests": {"storage": "2Gi" }}}}' -n tanzu-system-registry
persistentvolumeclaim/database-data-harbor-database-0 patched
 


Check status of pvc to confirm you see FileSystemResizePending example below shows info for database pvc. 

$ kubectl get pvc database-data-harbor-database-0 -n tanzu-system-registry -o yaml
conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-05-13T07:55:29Z"
    message: Waiting for user to (re-)start a pod to finish file system resize of
      volume on node.
    status: "True"
    type: FileSystemResizePending
 


Step 3 scale sts back to original state

$ kubectl scale sts/harbor-database --replicas=1 -n tanzu-system-registry 
  kubectl scale sts/harbor-redis --replicas=1 -n tanzu-system-registry

Step 4 check pvc and pod status

$ kubectl exec -it -n tanzu-system-registry harbor-database-0 -- df -h
Defaulted container "database" out of: database, data-migrator (init), data-permissions-ensurer (init)
Filesystem      Size  Used Avail Use% Mounted on
overlay          39G   18G   20G  47% /
tmpfs            64M     0   64M   0% /dev
tmpfs           512M  128K  512M   1% /dev/shm
/dev/sda2        39G   18G   20G  47% /etc/hosts
/dev/sdb        5.9G   65M  5.6G   2% /var/lib/postgresql/data
tmpfs           7.9G     0  7.9G   0% /proc/acpi
tmpfs           7.9G     0  7.9G   0% /proc/scsi
tmpfs           7.9G     0  7.9G   0% /sys/firmware


Step 5 To update harbor package, delete statefulset, first confirming that sts are present


kubectl get sts -n tanzu-system-registry

Step 6 Once cofirmed delete sts. 

 kubectl delete statefulset --cascade=orphan harbor-database -n tanzu-system-registry
 kubectl delete statefulset --cascade=orphan harbor-redis-0 -n tanzu-system-registry

Step 7 Update harbor-values.yaml databse pvc size consistent with current pvc size

persistence:
  persistentVolumeClaim:
    database:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 2Gi


 
Step 8 update harbor package, wait until success

$ tanzu package installed update harbor  -p harbor.tanzu.vmware.com -v 2.7.1+vmware.1-tkg.1 --values-file harbor-values.yaml  -n <namespace>
Check harbor-redis sts has been updated


$ kubectl get sts -n tanzu-system-registry harbor-database -o json | jq -r '.spec.volumeClaimTemplates[0].spec.resources'
{
  "requests": {
    "storage": "2Gi"
  }
}

Unset the Harbor ReadOnly mode

Login as admin => Navigate to Configuration => System Settings => Repository Read Only

Additional Information

Powered by Wolken Software