Unable to add labels in the kube-system namespace of TKG guest cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Trying to add labels in the kube-system namespace of TKG guest cluster fails and reconciles back.

Environment

vSphere with Tanzu

Tanzu Kubernetes release 1.30.x

Cause

While checking the kube-system namespace, found it managed by kapp controller app and the app id belongs to metric server.

# kubectl get namespace kube-system -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    kapp.k14s.io/delete-strategy: orphan
    kapp.k14s.io/identity: v1;//Namespace/kube-system;v1
    kapp.k14s.io/original: '{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{"kapp.k14s.io/delete-strategy":"orphan"},"labels":{"kapp.k14s.io/app":"1730764389988750191","kapp.k14s.io/association":"v1.95cec5f2fcd8d4a25497f4e01f997994"},"name":"kube-system"}}'
    kapp.k14s.io/original-diff-md5: ffc4598d1857ebb6a3cb3dcb4fa979ee
  creationTimestamp: "2024-11-04T23:52:28Z"
  labels:
    kapp.k14s.io/app: "1730764389988750191" ## <<== Referencing the App ID for the metrics-server.
    kapp.k14s.io/association: v1.95cec5f2fcd8d4a25497f4e01f997994
    kubernetes.io/metadata.name: kube-system
  name: kube-system
  resourceVersion: "10853894"
  uid: 78912389-0336-4bd9-9262-43d564046344
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Metrics-server.yaml:

Metrics-server yaml:

- apiVersion: kappctrl.k14s.io/v1alpha1
  kind: App
  metadata:
    annotations:
      packaging.carvel.dev/package-ref-name: metrics-server.tanzu.vmware.com
      packaging.carvel.dev/package-version: 0.6.2+vmware.3-tkg.6-vmware
... <<== snipped
  status:
    conditions:
    - status: "True"
      type: ReconcileSucceeded
    consecutiveReconcileSuccesses: 1899
    deploy:
      exitCode: 0
      finished: true
      kapp:
        associatedResources:
          groupKinds:
          - group: ""
            kind: Namespace
... <<== snipped
          - group: rbac.authorization.k8s.io
            kind: RoleBinding
          label: kapp.k14s.io/app=1730764389988750191 <<== App ID.
          namespaces:

Resolution

The metrics-server package should not take ownership of the kube-system namespace.

This will be fixed in future Tanzu Kubernetes releases 1.33