Attempting to expand Harbor Database component size fails and package reconciliation status goes into "not reconciled" state
Article ID: 376220
Updated On:
Products
Issue/Introduction
1. On expanding the Harbor Database component, after patching, the PVC shows new size (In the example below we expand the harbor-redis from 1Gi to 2Gi)
A. Check the current PVC size.
kubectl get pvc -n tanzu-system-registry data-harbor-redis
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-harbor-redis Bound pvc-<ID> 1Gi RWO default 36m
B. Check the current status of Harbor Package.
tanzu package installed list -n harbor
NAME PACKAGE-NAME PACKAGE-VERSION STATUS
harbor harbor.tanzu.vmware.com 2.7.1+vmware.1-tkg.1 Reconcile succeeded
C. Expand Harbor Redis PVC
kubectl patch pvc data-harbor-redis --patch '{"spec": {"resources": {"requests": {"storage": "<new-size>Gi" }}}}' -n harbor
persistentvolumeclaim/data-harbor-redis patched
D. Confirm it has expanded.
kubectl get pvc -n tanzu-system-registry data-harbor-redis
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-harbor-redis Bound pvc-<ID> 2Gi RWO default 42m
2. However, after modifying the Data values for the concerned database component and deleting the stateful set, the new package fails to reconcile.
A. Edit the data values for the redis-component using the vi editor.
vi harbor-data-values-mod.yaml
B. Delete the stateful set.
kubectl delete statefulset --cascade=orphan harbor-redis -n harbor
statefulset.apps "harbor-redis" deleted
C. Apply the updated component data values yaml file.
tanzu package installed update harbor -p harbor.tanzu.vmware.com -v <version> -f harbor-data-values-mod.yaml -n harbor
Updating installed package 'harbor'
Getting package install for 'harbor'
Getting package metadata for 'harbor.tanzu.vmware.com'
Updating secret 'harbor-harbor-values'
Updating package install for 'harbor'
Waiting for 'PackageInstall' reconciliation for 'harbor'
'PackageInstall' resource install status: ReconcileSucceeded
Updated installed package 'harbor' in namespace 'harbor'
D. Now when you check the current status of Harbor Package, it shows "Reconcile Failed".
tanzu package installed list -n harbor
NAME PACKAGE-NAME PACKAGE-VERSION STATUS
harbor harbor.tanzu.vmware.com <version> Reconcile failed: Error (see .status.usefulErrorMessage for details)
E. On running the tanzu package installed get command, you see the below error
tanzu package installed get harbor -n harbor
NAME: harbor
PACKAGE-NAME: harbor.tanzu.vmware.com
PACKAGE-VERSION: 2.7.1+vmware.1-tkg.1
STATUS: Reconcile failed: Error (see .status.usefulErrorMessage for details)
CONDITIONS: [
{ReconcileFailed True Error (see .status.usefulErrorMessage for details)}]
USEFUL-ERROR-MESSAGE:] Waited for 1.xs due to client-side throttling, not priority and fairness, request: GET:https://<ClusterIP-service>:443/apis/authentication.concierge.pinniped.tmc.cloud.vmware.com/v1alpha1
kapp: Error: update statefulset/harbor-redis (apps/v1) namespace: harbor:
Updating resource statefulset/harbor-redis (apps/v1) namespace: harbor:
API server says:
StatefulSet.apps "harbor-redis" is invalid: spec:
Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden (reason: Invalid)
Environment
vSphere CSI Driver
Cause
vSphere CSI driver does not support expansion of PV that is related to a pod created by Stateful Set either by online mode or offline mode.
Resolution
Resize of Stateful Set volume is still under development in Kubernetes and therefore not yet supported.
Additional Information
Feedback
