Unable to deploy Kubernetes Cluster: Cluster creation gets stuck after deploying one Control Plane and Worker node in VMware Cloud Director.

Products

• VMware Cloud Director

Issue/Introduction

• While deploying a Kubernetes Cluster, only one Control Plane and Worker Node is deployed and the cluster creation is stuck.
• Control Plane capvcd logs has the following error:
  •  ERROR   Failed to patch VCDCluster      {"controller": "vcdcluster", "controllerGroup": "infrastructure.cluster.x-k8s.io", "controllerKind": "VCDCluster", "VCDCluster": {"name":"<Cluster Name>","namespace":"<Cluster Name-ns>"}, "namespace": "<Cluster Name-ns>", "name": "<Cluster Name>", "reconcileID": "########-####-####-####-############", "error": "Operation cannot be fulfilled on vcdclusters.infrastructure.cluster.x-k8s.io \"<cluster name>\": the object has been modified; please apply your changes to the latest version and try again", "errorCauses": [{"error": "Operation cannot be fulfilled on vcdclusters.infrastructure.cluster.x-k8s.io \"<cluster name>\": the object has been modified; please apply your changes to the latest version and try again"}, {"error": "Operation cannot be fulfilled on vcdclusters.infrastructure.cluster.x-k8s.io \"<cluster name>\": the object has been modified; please apply your changes to the latest version and try again"}]}

Environment

VMware Cloud Director 10.6.x
VMware Cloud Director Container Service Extension 4.2.x

Cause

CSE server's component versions are not compatible with the deployed CSE version.

Resolution

Please upgrade the CSE Server's Component Version following the steps:

1. As a provider user, go to CSE UI plugin's CSE Management tab -> Server details -> UPDATE SERVER -> Patch Version Upgrade (documented here).
2. Patch version upgrade workflow will update these versions in the vcdkeconfig entity. Afterwards, check the vcdkeconfig entity to confirm that version values are updated, and confirm that any potential custom values such as registry are retained.
3. Power off and power on all CSE server vApps. This will allow the CSE server to fetch/cache the updated vcdkeconfig.
4. For your Kubernetes clusters, the component versions that you are using will now be outdated. Clusters must be updated manually using cluster upgrade script. That process is documented here.