Workload Cluster Creation Delays and Performance Issues Due to kapp-controller Timeout

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When creating a vSphere Kubernetes Workload Cluster, users observe significant delays (e.g., up to one hour) between the creation of the first Control Plane (CP) node and the subsequent Worker nodes or additional CP nodes.
• Supervisor tanzu-addons-controller-manager logs:
  

  HH:MM:SS 1 clusterbootstrap_controller.go:xxx] "unable to create or patch Package resource vmware-system-tkg/vsphere-cpi... on cluster: [cluster-name]" err="failed to get API group resources: unable to retrieve the complete list of server APIs: data.packaging.carvel.dev/v1alpha1: the server is currently unable to handle the request"

• Supervisor kube-apiserver logs:
  

  HH:MM:SS 1 controller.go:xxx] "Unhandled Error" err="loading OpenAPI spec for \"v1alpha1.data.packaging.carvel.dev\" failed with: failed to download v1alpha1.data.packaging.carvel.dev: resource not found"

Environment

vSphere Supervisor 8.0

vSphere Kubernetes releases v1.35.2

Cause

The issue is typically caused by a hung state or intermittent failure within the kapp-controller pod on the Supervisor Cluster. This results in the tanzu-addons-controller-manager being unable to retrieve the complete list of server APIs, specifically failing on v1alpha1.data.packaging.carvel.dev.

Resolution

To resolve this issue and restore normal cluster creation performance, restart the kapp-controller deployment within the Supervisor Cluster.

Option 1: Rolling Restart (Recommended)

Perform a rolling restart of the deployment to ensure a fresh pod is scheduled without manual deletion of individual pod names.

1. Log in to the Supervisor Cluster context.
2. Run the following command:
   

   kubectl rollout restart deployment kapp-controller -n vmware-system-appplatform-operator-system

Option 2: Direct Pod Deletion

If a rapid intervention is required for a specific hung pod:

1. Identify the pod name:
   

   kubectl get pods -n vmware-system-appplatform-operator-system | grep kapp-controller

2. Delete the pod:
   

   kubectl delete pod [kapp-controller-pod-name] -n vmware-system-appplatform-operator-system

Verification
Monitor the status of the new pod to ensure it reaches the Running state:

kubectl get pods -n vmware-system-appplatform-operator-system -w | grep kapp-controller

Once the kapp-controller is healthy, the tanzu-addons-controller-manager should resume the bootstrap process for the workload cluster immediately.