Partial creation of TKG cluster (1/3) from CSE with no error reported in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to create a Tanzu Kubernetes Grid (TKG) cluster from the VMware Cloud Director Tenant Portal, the cluster is only partially created.
  For example, if you configure the cluster with 3 worker nodes, it may complete with only 1 node deployed (1/3) and display no errors in the user interface.
  

     

• Upon investigating the worker node logs located at: \worker-node\worker-node\log\containers\rdeprojector-controller-manager-*.log or \worker-node\worker-node\log\pods\rdeprojector-system_rdeprojector-controller-manager-*\manager\X.log

• The following error is observed indicating an API version mismatch:

ERROR    Reconciler error    {"controller": "rdeprojector", "controllerGroup": "capvcd.cloud-director.vmware.com", "controllerKind": "RDEProjector", "rDEProjector": {"name":"test-smc","namespace":"default"}, "namespace": "default", "name": "test-smc", "reconcileID": "####-#####-#####-#####-####", "error": "Error creating VCD client to reconcile RDE [urn:vcloud:entity:vmware:capvcdCluster:####-#####-#####-#####-####]: unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set authorization header: [error finding LoginUrl: could not find valid version for login: API version 36.0 is not supported: version = 36.0 is not supported]]]", "errorVerbose": "unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set authorization header: [error finding LoginUrl: could not find valid version for login: API version 36.0 is not supported: version = 36.0 is not supported]]]\nError creating VCD client to reconcile RDE [urn:vcloud:entity:vmware:capvcdCluster:####-#####-#####-#####-####]\ngitlab.eng.vmware.com/core-build/vcd-rde-projector/controllers.(*RDEProjectorReconciler).reconcileNormal\n\t/build/mts/release/

 

Environment

VMware Cloud Director 10.6.X
Container Service Extension(CSE) 4.2.X

Cause

This issue occurs because the installed version of Container Service Extension (4.2.x) is not compatible with the installed version of VMware Cloud Director (10.6.x). The API version 36.0 utilized by VMware Cloud Director 10.6.x is not supported by CSE 4.2.x, causing the cluster reconciliation to fail silently in the UI while logging the API version error in the background.

Resolution

   • Log in to the VMware Cloud Director Provider Portal.
   •Navigate to Kubernetes Container Cluster.
   •Select the option to update the CSE Server.
   •Change the RDE Project Version to 0.7.1.

     

   •Click Save to apply the changes.
   •Re-deploy the CSE Cluster.

Note: The cluster will deploy successfully, but it may take 20 to 30 minutes for the updated cluster status to fully reflect in the UI.