VKS Guest Cluster Creation or Upgrade Hangs Due to Expired Carvel Packaging Certificate
search cancel

VKS Guest Cluster Creation or Upgrade Hangs Due to Expired Carvel Packaging Certificate

book

Article ID: 385432

calendar_today

Updated On:

Products

Tanzu Kubernetes Runtime vSphere with Tanzu

Issue/Introduction

  • Tanzu Kubernetes Grid (TKG) Guest Cluster (VKS) deployments or upgrades remain in a "Creating" or "Updating" state indefinitely. Reconciliation fails during preflight checks because control plane nodes cannot reach a "Ready" status and addon packages are never installed.
  • Deployment will seem to be stuck on creating phase:

Waiting for control plane to pass preflight checks to continue reconciliation: [machine <name> does not have APIServerPodHealthy condition, machine <name> does not have ControllerManagerPodHealthy condition, machine <name> does not have SchedulerPodHealthy condition, machine <name> does not have EtcdPodHealthy condition, machine <name> does not have EtcdMemberHealthy condition]

  • Running kubectl get machines -n <namespace> will show machines deployed but no node name assigned.
  • Cluster would still be waiting on the packages to be deployed by kapp which will eventually cause the Nodes to be marked as Ready.
  • After opening an SSH session into control plane of guest cluster will reveal nodes in not ready state:
root@<control plane>:~# kubectl get nodes
NAME                                 STATUS     ROLES           AGE   VERSION
<cluster name>-1-2c42d-849c649pv67   NotReady   <none>          27m   v1.24.9+vmware.1
<cluster name>-1-2c42d-849c64f96lm   NotReady   <none>          27m   v1.24.9+vmware.1
<cluster name>-1-2c42d-849c64k8998   NotReady   <none>          27m   v1.24.9+vmware.1
<cluster name>-vz9dt-qrr4p           NotReady   control-plane   29m   v1.24.9+vmware.1
  • None of the addon packages will be installed due to which the Nodes never move into a Ready state:
root@<control plane>:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-#####b5d7-##### 0/1 Pending 0 20m
kube-system coredns-#####88cf-##### 0/1 Pending 0 20m
kube-system coredns-#####88cf-##### 0/1 Pending 0 20m
kube-system docker-registry-<cluster name>-<pod id> 1/1 Running 0 17m
kube-system docker-registry-<cluster name>-<pod id> 1/1 Running 0 17m
kube-system docker-registry-<cluster name>-<pod id> 1/1 Running 0 17m
kube-system docker-registry-<cluster name> 1/1 Running 0 20m
kube-system etcd-<cluster name> 1/1 Running 0 20m
kube-system kube-apiserver<cluster name>1/1 Running 0 20m
kube-system kube-controller-manager-<cluster name> 1/1 Running 0 20m
kube-system kube-proxy-##### 1/1 Running 0 18m
kube-system kube-proxy-##### 1/1 Running 0 18m
kube-system kube-proxy-##### 1/1 Running 0 18m
kube-system kube-proxy-##### 1/1 Running 0 20m
kube-system kube-scheduler-<cluster name> 1/1 Running 0 20m

YYYY-MM-DDTHH:MM:SSZ stderr F I0106 10:28:38.785657 1 clusterbootstrap_controller.go:170] ClusterBootstrapController "msg"="Reconciling cluster" "cluster-name"="<cluster name>" "cluster-ns"=<cluster namespace>"
YYYY-MM-DDTHH:MM:SSZ stderr F I0106 10:28:38.785705 1 clusterbootstrap_controller.go:389] ClusterBootstrapController "msg"="ClusterBootstrap for cluster <cluster name>/<cluster namespace> does not exist, creating from template vmware-system-tkg/v1.24.9---vmware.1-tkg.4" "cluster-name"="<cluster name>" "cluster-ns"=<cluster namespace>"
YYYY-MM-DDTHH:MM:SSZ stderr F E0106 10:28:38.790869 1 clusterbootstrapclone.go:491] ClusterBootstrapController "msg"="unable to fetch Package.Spec.RefName or Package.Spec.Version from Package <cluster name>/antrea.tanzu.vmware.com.1.7.2+vmware.1-tkg.1-advanced" "error"="the server is currently unable to handle the request (get packages.data.packaging.carvel.dev antrea.tanzu.vmware.com.1.7.2+vmware.1-tkg.1-advanced)"
YYYY-MM-DDTHH:MM:SSZ stderr F E0106 10:28:38.790888 1 clusterbootstrapclone.go:434] ClusterBootstrapController "msg"="unable to clone secrets or providers" "error"="the server is currently unable to handle the request (get packages.data.packaging.carvel.dev antrea.tanzu.vmware.com.1.7.2+vmware.1-tkg.1-advanced)"
YYYY-MM-DDTHH:MM:SSZ stderr F E0106 10:28:38.790922 1 controller.go:317] controller/cluster "msg"="Reconciler error" "error"="the server is currently unable to handle the request (get packages.data.packaging.carvel.dev antrea.tanzu.vmware.com.1.7.2+vmware.1-tkg.1-advanced)" "name"="<cluster name>" "namespace"="<cluster namespace>" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"
  • This shows that the ClusterBootstrap object which is responsible for the addon packages to be installed on the TKC has not been created due to Package Custom Resource Definitions (CRDs) reconcile issues.
  • For guest clusters: kubectl get pkgi -A (Packages often show in Failed state due to this issue).
  • The API server logs have the following entries:
YYYY-MM-DDTHH:MM:SSZ stderr F E0102 14:58:28.959734       1 controller.go:113] loading OpenAPI spec for "v1alpha1.data.packaging.carvel.dev" failed with: Error, could not get list of group versions for APIService
YYYY-MM-DDTHH:MM:SSZ stderr F I0102 14:58:28.960106       1 controller.go:126] OpenAPI AggregationController: action for item v1alpha1.data.packaging.carvel.dev: Rate Limited Requeue.
YYYY-MM-DDTHH:MM:SSZ stderr F E0102 14:58:28.960470       1 controller.go:116] loading OpenAPI spec for "v1alpha1.data.packaging.carvel.dev" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: x509: certificate has expired or is not yet valid: current time YYYY-MM-DDTHH:MM:SSZ is after YYYY-MM-DDTHH:MM:SSZ
YYYY-MM-DDTHH:MM:SSZ stderr F , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
  • Check kapp-controller uptime on the Supervisor: kubectl get pods -A | grep -i kapp-controller (Note if AGE > 365d):
vmware-system-appplatform-operator-system   kapp-controller-#######566d-#####                                  2/2     Running            0                 483d
  • Executing below command confirms expiry of certificate:
kubectl get apiservice v1alpha1.data.packaging.carvel.dev -o jsonpath='{.spec.caBundle}' | base64 -d | openssl x509 -text -noout
 
Serial Number: 2 (0x2)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kapp-controller-ca@173####980
        Validity
            Not Before: MM  DD HH:MM:SS YYYY GMT
            Not After : MM  DD HH:MM:SS YYYY GMT
        Subject: CN=kapp-controller@173####980


Cause

The kapp-controller uses a self-signed CA for the v1alpha1.data.packaging.carvel.dev APIService. This certificate has a limited validity period (typically 1 year). If the kapp-controller pod has been running longer than the certificate validity (e.g., >365 days), the certificate expires. This prevents the Supervisor Cluster from reconciling Package and ClusterBootstrap resources required for guest cluster lifecycle operations.

Resolution

Restart the kapp-controller pod to trigger a rotation and renewal of the internal certificates.

1. Identify the kapp-controller pod name:

kubectl -n vmware-system-appplatform-operator-system get pods

2. Delete the pod to force a restart:

kubectl -n vmware-system-appplatform-operator-system delete pod kapp-controller-###

3. Confirm certificate renewal after restart of kapp controller pod:

kubectl get apiservice v1alpha1.data.packaging.carvel.dev -o jsonpath='{.spec.caBundle}' | base64 -d | openssl x509 -text -noout

The ClusterBootstrap and Package resources should now reconcile, allowing the Guest Cluster nodes to install addons and transition to the Ready state.

Powered by Wolken Software