VMware Cloud Foundation Installer failing at 71% on step "Deploy and configure the operations appliance"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• In the workflow, the specific subtask that is failing is -

Retrieve the status for VCF Operations with VCF Operations collector Deployment request

• The failure error observed is - 

Request createenvironment failed with error cause [{"messageId":"LCMVROPSYSTEM25001","message":"Operations initializing cluster failure","eventId":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","retry":true,"exceptionMessage":"Operations cluster configurations failed","exceptionStackTrace":"com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.ClusterInitException: Cluster configuration failed with status INITIALIZATION_FAILED. Check the the status of the cluster using /casa/cluster/status. if cluster is in INITIALIZED or INITIALIZING state then retry the request. if cluster configuration is failed , follow below steps\n 1. Check the casa logs of Operations or navigate to admin UI of Operations and fix the issue manually. 2. Once the issue is fixed , run the following command on master node: 'service vmware-casa restart' 3. Once master node services are up and running , retry the request from vRSLCM\n\tat com.vmware.vrealize.lcm.plugin.core.vrops.tasks.CheckNodeStatusTask.retry(CheckNodeStatusTask.java:302)\n\tat com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:60)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n","localizedMessageId":"VROPS_CHECKNODESTATUSTASK_ERROR_00006","parameters":null,"properties":{}}] Reference Token: XXXXXX

• The VCF Operations primary node casa cluster status state observed (by launching https://ops_primary_node_fqdn/casa/cluster/status) is null -

{"nodes_states": [{"name":"ops_primary_node_hostname", "address": "ops_primary_node_fqdn", "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","role":"MASTER","state":"CONFIGURED"}, {"name": "ops_secondary_node_hostname", "address":"ops_secondary_node_fqdn","uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","role":"REPLICA","state":"CONFIGURED"}, {"name": "ops_tertiary_node_hostname", "address": "ops_tertiary_node_fqdn", "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","role": "DATA", "state": "CONFIGURED"}], "cluster_state": null}

• The VCF Operations primary node UI > System Status > Cluster Status is "Online - Waiting to finish cluster expansion"

 

Environment

VCF 9.0.2

Cause

The VCF Operations cluster has failed to be initialized, this can be due to a number of factors.

Resolution

Please open a support case with Broadcom Technical Support to have the issue resolved.