VCFA deployment fails with LCMVSPHERECONFIG100009 timed out waiting for the condition on kubernetesclusters/vcf-mgmt-#######

Products

• VCF Automation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When deploying VCF Automation from SDDC Manager / Lifecycle, it fails due to a timeout waiting on the kubernetes cluster.
• VMSP template is successfully copied to the destination vCenter.
• No activity occurs in vCenter after the initial copy of the template.
• After approximately 2+ hours, the task fails with an error in SDDC Manager / Lifecycle:

  Retrieve the status of VCF Automation Deployment request - Failed - Unable to get request status for request <Request_ID> Reference Token: <Unique_Token_Number>

• Your destination vCenter is configured with 10.244.0.0/16.

Environment

• VCF SDDC Manager 9.0.x
• VCF Automation 9.0.x
• VCF Fleet Management 9.0.x

Cause

During deployment, SDDC-Manager/ Lifecycle will copy the VMSP template to vCenter server, then deploy a kind cluster on the Fleet Management server to facilitate the installation of VCF Automation on the destination vCenter / deployed VMSP appliance(s).

This cluster is configured using 10.244.0.0/16 which can be a duplicate of other resources in the environment (such as vCenter) causing a conflict.

Resolution

We can workaround the issue by updating the bootstrap.sh file with the pod networking details. Follow the below steps:

VCF Automation 9.0.x uses a fixed subnet (10.244.0.0/16) for deployment configurations. To use a different subnet, add a networking section to the bootstrap.sh file:

1. Secure a snapshot of the Fleet Management before proceeding with next steps.
2. SSH into the Fleet Management appliance as root.
3. Backup the file using: 

   cp /data/vmsp/scripts/bootstrap.sh /data/vmsp/scripts/bootstrap.sh.bkp

4. Edit the file: 

   vi /data/vmsp/scripts/bootstrap.sh

5. Locate the correct section (as illustrated below) and add the following two networking lines:

   # Create Kind config
   # containerd will forward traffic for the registry to the nodePort service
   # A similar configuration will be set on the new cluster through kubeadm in the vmsp-configs chart
   cat <<EOF >"${KIND_CONFIG}"
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   networking:                                             <--------------------- Update the following.
     podSubnet: "10.255.0.0/16"                            <--------------------- Update this line too. Mind the spacing.
   kubeadmConfigPatches:
   - |
     apiVersion: kubeadm.k8s.io/v1beta3

6. Save the file using :wq!
7. Re-attempt the deployment (either through SDDC-Manager / Lifecycle or from your initial VCF 9.0.x deployment appliance).

Additional Information

Should there be an attempt to deploy VCF Automation directly from Fleet Management, following KB should help to skip VCF-A deployment workflow in the VCF Installer UI: VCF Installer workflow task fails at VCF Automation deployment with 404 error