Harbor extension using an existing PersistentVolumeClaim(PVC) fails to deploy in Tanzu Kubernetes Grid v1.3.1

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

The purpose of this article is to provide a workaround around a failure encountered when the user is trying to deploy Harbor extension on a TKG v1.3.1 cluster with a modification to use an existing PersistentVolumeClaim (PVC) for the harbor-registry deployment.

Symptoms:
Pre-checks

• You are using tanzu CLI v1.3.1
• You are using TKG extension manifests from tkg-extensions-manifests-v1.3.1+vmware.1.tar.gz
• You are following instructions from v1.3.1 documentation to deploy Harbor Extension
• You want to use an existing persistent volume in your harbor deployment
• You have modified the configurations for property persistence in harbor-data-values.yaml file. For example:

persistence:
  persistentVolumeClaim:
    registry:
      existingClaim: "tkm-harbor-sandbox-1-pvc"
 

Symptom 1

Reconciliation failed after applying the harbor-extension.yaml to a TKG cluster. You can be verify this using the following:
kubectl -n tanzu-system-registry get app harbor
NAME     DESCRIPTION                                  SINCE-DEPLOY   AGE
harbor   Reconcile failed: Deploying: exit status 1   12s            13h

Symptom 2

Obtaining the stderr output from the app status property shows syntax issues as highlighted in the output below
Note - The output below also hints towards the name of the existing PVC you wanted to use with the harbor-registry deployment

kubectl -n tanzu-system-registry get app harbor -ojson | jq -r '.status.deploy.stderr'

kapp: Error: Applying update deployment/harbor-registry (apps/v1) namespace: tanzu-system-registry:
  Updating resource deployment/harbor-registry (apps/v1) namespace: tanzu-system-registry:
    Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Volumes: []v1.Volume: v1.Volume.VolumeSource:
      PersistentVolumeClaim: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|meClaim":"tkm-harbor|..., bigger context ...|,{"name":"registry-data","persistentVolumeClaim":"tkm-harbor-sandbox-1-pvc"},{"name":"registry-inter|... (reason: BadRequest)

Cause

This issue is caused because of a syntax error in the harbor-registry deployment YAML.

• The deployment gets scheduled to be deployed by the kapp controller using the configurations present in tkg-extensions-v1.3.1+vmware.1/extensions/registry/harbor/harbor-extension.yaml
• Once ytt renders the required Kubernetes objects to be deployed on the cluster, the malformed deployment YAML generated for harbor-registry deployment looks like the following (content is omitted for brevity & only shows the volumes section of the pod spec that gets embedded into the deployment spec)

Erroneous state
volumes:
- name: registry-data
  persistentVolumeClaim: <claim-name>

Resolution

Currently, there is no resolution and a possible fix in the product will be targeted for a future release. However, you can check out the workaround to get your use case in a working state and at the same time, have the kapp controller continue to manage the extension so reconciliation is successful each time the extension(s) are reconciled in the cluster.

Workaround:
You will see the erroneous state in the cluster from the Cause section of this article. And you want to get to the following correct state to get the Harbor extension deployed successfully with an existing PVC:

Correct state
- name: registry-data
  persistentVolumeClaim:
    claimName: <claim-name>

In order to achieve the correct state, you can execute the following instructions:

• Delete the Harbor App
  • kubectl delete -f harbor-extension.yaml 
  • kubectl delete app harbor -n tanzu-system-registry
• Do not delete harbor-data-values secret, namespace, existing PVC, and/or roles created as part of the installation.
• Once the application is deleted, update your harbor-extension.yaml file to reflect the following changes:

# harbor k14s objects managed by kapp-controller
---
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: harbor
  namespace: tanzu-system-registry
  annotations:
    tmc.cloud.vmware.com/managed: "false"
spec:
  syncPeriod: 5m
  serviceAccountName: harbor-extension-sa
  fetch:
    - image:
        url: projects.registry.vmware.com/tkg/tkg-extensions-templates:v1.3.1_vmware.1
  template:
    - ytt:
        ignoreUnknownComments: true
        paths:
          - tkg-extensions/common
          - tkg-extensions/registry/harbor
        inline:
          paths:
            update-registry-pvc.yaml: |
              #@ load("@ytt:overlay", "overlay")
              #@overlay/match by=overlay.subset({"kind": "Deployment", "metadata": {"name": "harbor-registry"}})
              ---
              spec:
                template:
                  spec:
                    volumes:
                      #@overlay/match by="name"
                      #@overlay/replace
                    - name: registry-data
                      persistentVolumeClaim:
                        claimName: "<UPDATE_CLAIM_NAME_HERE>"
          pathsFrom:
            - secretRef:
                name: harbor-data-values
  deploy:
    - kapp:
        rawOptions: ["--wait-timeout=5m"]

• Once the changes are updated, apply the harbor-extension.yaml file to the cluster to deploy the harbor extension
  • kubectl apply -f harbor-extension.yaml 
• Verify/Validate the status of the app
  • kubectl -n tanzu-system-registry get app harbor
  • kubectl -n tanzu-system-registry describe app harbor

Additional Information

Impact/Risks:
This issue impacts the ability to deploy harbor extension in TKG v1.3.1 using an existing PVC