Supervisor upgrade stuck after upgrading to 9.0.2 error "vmware-system-vks-public user=system:serviceaccount:svc-tkg-domain-:runtime-extension- pkg-sa: prohibited operation on system namespace"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

VKS version 3.5 failing to reconcile after upgrading VCF to 9.0.2 when using a private container registry. 

Error from supervisor shows below error 

Configured Control Plane VMs

Configured Core Supervisor Services
Service: tkg.vsphere.vmware.com. Reason: "ReconcileFailed".
Message: kapp: Error: waiting on reconcile packageinstall/runtime-extension (packaging.carvel.dev/v1alpha1) namespace: svc-tkg-domain-c#: Finished waiting unsuccessfully: Reconcile failed:
message: kapp: Error: update namespace/vmware-system-vks-public (v1) cluster: Updating resource namespace/vmware-system-
vks-public (v1) cluster: API server says: admission webhook "validate.namespace.appplatform.vmware.com" denied the request:
op=UPDATE namespace=vmware-system-vks-public user=system:serviceaccount:svc-tkg-domain-c#:runtime-extension-pkg-sa: prohibited operation on system namespace (reason:Forbidden).

Environment

VCF 9.0.1
9.0.2
vSphere Kubernetes Service

Cause

Issue cause due to packages on the registry not being signed correctly.

Checking applatform logs from the supervisor, confirms issue d

kubectl logs -n vmware-system-appplatform-operator-system -l control-plane=controller-manager

1 first party trust.go:115] "msg"="Package bundle is not trusted" "serviceID"="tkg" "verification"="tkg. 3.5.0-signature-verification-4ngpw" "version"="3.5.0+v1.34"
first party trust.go:204] "msg"="untrusted service" "namespace"="svc-tkg-domain-c9" "serviceID"="tkg" "version"="3.5.0+v1.34"

 

From supervisor release (9.0.0.0100) to ensure all Supervisor Service shipped by VMware has the correct signature to trust.

After upgrade if the VKS versions is higher than 3.4.0,  the Supervisor services are no longer whitelisted. This means it will block any services that don't have a valid signature. 

When packages are pushed to registry --cosign-signature flag needs to be used. 

 

Resolution

Remove and re-upload images to registry using the cosign flag

Generate a tar binary of the imgpkg  bundle.

imgpkg copy --tar tkg-service-v<version>.tar --to-repo ${REGISTRY_URL}/tkg-service --cosign-signatures --registry-ca-cert-path ca.crt


Upload the imgpkg bundle to your private container image registry.

imgpkg copy --tar tkg-service-v<version>.tar --to-repo ${REGISTRY_URL}/tkg-service --cosign-signatures --registry-ca-cert-path ca.crt

Restart applatform deployments to resync packages

kubectl rollout restart deployment -n vmware-system-appplatform-operator-system

 

Steps on updating packages to private registry. 
Upgrade VKS from a Private Registry