Manual volume expansion for Harbor "MISCONF Redis..." failure in vSphere with Tanzu 7.0

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

The embedded Harbor registry becomes inaccessible, resulting in HTTP 500 Internal Server Errors and health check failures.
Dependent Harbor pods (such as jobservice and registry) enter a CrashLoopBackOff state or fail with a connection refused error.
Checking the events and logs of the harbor-redis pod reveals the following errors:

MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk.

Furthermore, attempting to directly patch the PVC to expand capacity in this state will be rejected with an error indicating that the namespace quota has been exceeded.

Error from server (Forbidden): persistentvolumeclaims "<pvc-name>" is forbidden: exceeded quota: <namespace>-storagequota, requested: ... used: ... limited: ...

Environment

VMware vSphere with Tanzu 7.0.x

Cause

The Persistent Volume (PV) backing the Harbor Redis pod has reached 100% capacity, preventing Redis from saving background snapshots (bgsave).
This triggers the MISCONF state, disabling further write operations.
In vSphere 7.0, the implementation does not allow expanding the storage limit of the embedded Harbor registry from the vSphere Client UI, preventing standard offline PVC expansion operations.

Resolution

Warning: This procedure scales down the Harbor Redis StatefulSet, which will temporarily interrupt access to the Harbor registry.

1. Refer to "Troubleshooting vSphere Supervisor Control Plane VMs"  and SSH into the supervisor control plane VM.
2. Identify the Harbor registry-specific namespace name and the Redis pod name.
   

   kubectl get pods -A -l component=redis
   出力例：
   NAMESPACE                         NAME                             READY   STATUS    RESTARTS   AGE
   vmware-system-registry-349###19   harbor-349###19-harbor-redis-0   1/1     Running   0          9d

3. Check the ResourceQuota name associated with the namespace.
   

   kubectl get resourcequota -n <harbor-namespace name> 
   Example output: 
   NAME AGE REQUEST LIMIT 
   vmware-system-registry-349###19-storagequota    10d vsan-default-storage-policy.storageclass.storage.k8s.io/requests.storage: 200Gi/200Gi

4. Edit ResourceQuota to temporarily increase the storage limit.
   Once the editor opens, find the hard limit for the storage class you are using (e.g., vsan-default-storage-policy.storageclass.storage.k8s.io/requests.storage) and increase the value (e.g., from 200Gi to 220Gi). 
   

   kubectl edit resourcequota <ResourceQuota name> -n <harbor-namespace name> 
   Example output: 
      ... (omitted) 
      spec: 
        hard: 
          vsan-default-storage-policy.storageclass.storage.k8s.io/requests.storage: 200Gi *Target for change 
      status: 
        hard: 
          vsan-default-storage-policy.storageclass.storage.k8s.io/requests.storage: 200Gi *Target for change 
        used: 
          vsan-default-storage-policy.storageclass.storage.k8s.io/requests.storage: 200Gi *The 200Gi in used is not included 
   
   After making changes, press Esc, then Shift + zz to save and exit.

5. Check the name of the Harbor Redis StatefulSet to be expanded.
   

   kubectl get sts -n <harbor-namespace name> -l component=redis 
   Example output: 
   NAME READY AGE 
   harbor-349###19-harbor-redis    1/1 10d

6. To safely unmount the volume, we scale down the Harbor Redis StatefulSet replicas to 0.

   kubectl scale sts <harbor-redis-statefulset名> -n <harbor-namespace名> --replicas=0

7. Identify the name of the Persistent Volume Claim (PVC) to be extended.
   

   kubectl get pvc -n <harbor-namespace名> -l component=redis
   出力例: 
   NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE 
   data-harbor-349###19-harbor-redis-0   Bound pvc-9aaa8930-493e-4957-9b42-026b9###1f3d 6G RWO vsan default storage policy 10d

8. Patch Redis's Persistent Volume Claim (PVC) to increase the claim size (e.g., from 6Gi to 18Gi).

   kubectl patch pvc <redis-pvc名> -n <harbor-namespace名> -p '{"spec":{"resources":{"requests":{"storage":"18Gi"}}}}'

9. Verify that the PVC status is FileSystemResizePending.

   Run `kubectl get pvc <redis-pvc name> -n <harbor-namespace> -o yaml | grep FileSystemResizePending -C 2` 
   and verify that the result includes `FileSystemResizePending`.

10. To remount the volume and complete the file system expansion, scale up the Harbor Redis StatefulSet replica to 1.
    

    kubectl scale sts <harbor-redis-statefulset名> -n <harbor-namespace名> --replicas=1

11. Verify that the size of the Redis pod /var/lib/redis has increased to 18GB.
    *Use the Redis pod name confirmed in step 2.
    

    kubectl exec -it <harbor-redis-pod名> -n <harbor-namespace名> -- df -h
    出力例：
    Filesystem     Size  Used Avail Use% Mounted on
    overlay        253M   13M  239M   6% /
    tmpfs           64M     0   64M   0% /dev
    shm             64M     0   64M   0% /dev/shm
    /dev/sda       253M   13M  239M   6% /dev/termination-log
    /dev/sdb        18G   48K   18G   1% /var/lib/redis

12. Verify that the Redis pod has started successfully and that the connection to Harbor has been restored.

Additional Information

Note: For vSphere 8.x environments, please follow the instructions for Expansion for registry PV for Harbor as as supervisor service.

• vSphere with Tanzu 7.0 における「MISCONF Redis is configured to save RDB snapshots」による Harbor 障害と手動容量拡張手順