VKS guest cluster creation is stuck at Control Plane in Provisioned state

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• VKS guest cluster gets stuck during creation.
• The first control plane gets deployed and is at provisioned state and does not proceed further.
• When checking the pkgi for the affected guest cluster, the kapp-controller shows the below error:
  
  $ kubectl get pkgi -n <namespace>
Reconcile failed: Error (see .status.usefulErrorMessage for details)


• When describing the same kapp-controller, the below error exists:
  
  $ kubectl describe pkgi <kapp-controller-name> -n <namespace>
    Useful Error Message:    kapp: Error: waiting on reconcile deployment/kapp-controller (apps/v1) namespace: tkg-system:
  Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "kapp-controller-#######" has timed out progressing.))


• When checking the logs of the ako pods in the supervisor, it shows the below errors:
  
  YYYY-MM-DDThh:mm:ss rest/rest_operation.go:###      key: admin/domain-c##-<cluster-name>-control-plane-service, msg: Error in rest operation is not of type AviError, err: Rest request error, returning to caller, *errors.errorString
hh:mm:ss avisession.go:###] CheckControllerStatus is disabled for this session, not going to retry.
YYYY-MM-DDThh:mm:ss rest/dequeue_nodes.go:###   key: admin/domain-c##-<cluster-name>-control-plane-service, msg: got error while executing rest request: Error during PUT: Rest request error, returning to caller, adding to slow retry queue

Environment

• vCenter 8.x
• vCenter 9.x

Cause

This is most likely caused by low memory in the AVI load balancer Service Engine.

Resolution

Increase the memory for the AVI load balancer's Service Engine.

Check the below document for information regarding increasing the memory:

SE Memory Consumption