VCF Bringup fails during vCenter Server Deployment with vLCM

Products

• VMware Cloud Foundation
• VMware Cloud Foundation 5.x

Issue/Introduction

• This article can be used by customers in order to overcome the failure and proceed with the management domain deployment.
• VCF Management domain deployment (Bringup) on Cloud Builder appliance fails during vCenter Server deployment with the below error when vLCM is enabled (in XLS property "Enable vLCM Cluster Image" is set to Yes, or if using JSON the field "clusterImageEnabled" under "clusterSpec" has value of "true"):

  File "/build/mts/release/bora-21930385/bora/install/vcsa-installer/vcsaCliInstaller/cli_tasks/vlcm_cluster_creation_tasks/vlcm_cluster_creation_postdeploy.py", line 220, in execute AttributeError: 'NoneType' object has no attribute

Environment

VMware Cloud Foundation 5.1.0
VMware Cloud Foundation 5.1.1

VMware Cloud Foundation 5.2

Cause

The failure is caused by a race condition in vCenter Server installer which fails to create a datacenter once the vCenter Server is deployed.

Resolution

Currently there is no resolution to the issue. This will be fixed in the future release.

 

• Workaround 1 - Switch to VUM instead of vLCM based
  
  The failed deployment can be continued if you switch the cluster configuration from vLCM to VUM. In order to so, follow this KB Failed bringup via the Cloud Builder - Process to restart the workflow via API and in step 1. from the Resolution section apply the following changes to the input:

• • If using XLS, change the following in "Deploy Parameters" sheet:
    • Change "Enable vLCM Cluster Image" to No
    • Change "Enable vSAN-ESA" to No
  • If using JSON, change the values of the following fields:
    • "clusterImageEnabled": false (this is under "clusterSpec")
    • Delete the "esaConfig" element under "vsanSpec" or modify it to have "enabled": false
      
      
• Workaround 2 - To deploy vLCM based management cluster

It is also possible to continue the deployment with vLCM (and VSAN ESA if needed) with some manual steps:

1. 1. Bringup fails at vCenter deployment step and vCenter Virtual Machine is deleted as part of the undo step. In order to keep the vCenter Virtual Machine follow the below steps
      1. SSH to Cloudbuilder VM with admin and su to root
      2. Edit application.properties file

         vi /opt/vmware/bringup/webapps/bringup-app/conf/application.properties

      3. Add property

         orchestrator.task.stopOnFirstFailure=true

      4. Restart Bringup service

         systemctl restart vcf-bringup

   2. Retry bringup using CloudBuilder UI, if vCenter deployment fails again, the vCenter Virtual Machine will not get deleted but it will be fully functional.
   3. Login to vCenter UI and create a Datacenter with the same name provided in Bringup input.
   4. Create a cluster:
      1. Start the New cluster wizard and use the same name provided in Bringup input.
      2. Leave vSphere DRS and vSphere HA unselected.
      3. Check the vSAN option and select "Enable vSAN ESA" checkbox if this was selected in Bringup input.
      4. Leave "Manage all hosts in the cluster with a single image" option selected in order to configure VLCM
      5. Select "Import image from a new host" radio button
      6. Leave "Manage configuration at a cluster level" option unselected and click Next.
      7. On the "Image" page of the wizard populate the details (FQDN and credentials) of the first host used for Bringup (the one that is currently hosting the vCenter Virtual Machine)
      8. Click "Find host" and accept the host certificate.
      9. Leave "Also move selected host to cluster" option selected.
      10. Click Next, review the settings and finish the wizard.
   5. Obtain pem certificate from the host by executing the following on the CB using root user (su):

      openssl s_client -showcerts -connect <esxi-fqdn>:443 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > esx.pem

   6. Delete the existing host certificate from CB trust store:

      keytool -delete -alias esxi-fqdn -cacerts  -storepass changeit

   7. Import the host certificate into CB trust store by executing:

      keytool -import -alias esx -keystore /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts -file esx.pem -storepass changeit

   8. After the cluster creation task in vCenter UI is completed retry Bringup deployment from CloudBuilder UI.