Error: "VCF Operations Fleet Management is Not Ready" when navigating to VCF Management in VCF Operations 9.x

Products

VCF Operations

Issue/Introduction

When you navigate to VCF Operations > Fleet management > Lifecycle > VCF management, you encounter a "VCF Operations Fleet Management is Not Ready" error. This prevents lifecycle management operations within the VCF environment.
During this time, the Fleet Lifecycle services remain operational without health alerts. However, an inventory sync attempt fails with the following error:

Error Code: LCMVROPCONFIG20066
Operations Config Error: Unknown Host Exception. Retry with correct hostname. If hostname is correct, ensure either Operations DNS server is present in VCF Operations Fleet Management or Operations entries are present in the DNS server of VCF Operations Fleet Management. Operations master node address provided is not resolvable null
A secondary error may also be observed during capability pushes:

Error Code: LCMVRLICONFIG40119
Error occurred while pushing capabilities to VCF Ops Logs, statusCode : 400 : {"errorMessage":"Some fields have incorrect values","errorCode":"FIELD_ERROR","errorDetails":{"oidcConfigUrl":[
{"errorCode":"com.vmware.loginsight.api.errors.field_required","errorMessage":"Value cannot be empty."}
The following log entries were observed in /var/log/vrlcm/vmware_vrlcm.log on the Fleet Management Appliance:

ClusterStatus returned data 200 <- queried <Ops_Node_Master_FQDN>
<Ops_Node_Replica_FQDN> role=REPLICA <Ops_Node_MASTER_FQDN> role=MASTER <Ops_Node_data_FQDN> role=DATA
ClusterStatus returned data 200 <- queried <Ops_Node_Replica_FQDN>
<Ops_Node_Replica_FQDN> role=REPLICA <Ops_Node_Master_FQDN> role=DATA <Ops_Node_data_FQDN> role=DATA
^^^^ MASTER lost
Running the query https://<Ops_Node_FQDN>/casa/cluster/status across all Analytics nodes confirms that the current master node is missing the MASTER role

"nodes_states": [
"name": "<Ops_Node_Replica_FQDN>",
"address": "<Ops_Node_Replica_FQDN>",
"uuid": "<Nodeidentifier>",
"role": "REPLICA",
"state": "CONFIGURED"
},
"name": "<Ops_Node_Master_FQDN>",
"address": "<Ops_Node_Master_FQDN>",
"uuid": "<Nodeidentifier>",
"role": "DATA",
"state": "CONFIGURED"
"name": "<Ops_Node_Data_FQDN>",
"address": "<Ops_Node_Data_FQDN>",
"uuid": "<Nodeidentifier>",
"role": "DATA",
"state": "CONFIGURED"
"cluster_state": "INITIALIZED"
An inspection of the cachedRoles.json file, as outlined in KB 368959, reveals that the ADMIN role is missing on the current master node.
Expected Configuration: All other nodes correctly display the full set of primary roles (ADMIN, DATA, UI).
Actual Configuration: The current master node (<Current_master_Node-FQDN>) is missing the ADMIN role and only lists DATA and UI.
Example Output (Other Nodes):
```
"cachedRoles": [
    "ADMIN",
    "DATA",
    "UI"
],
"nodeIdentifier": "#####-####-####-####-#############"
```
Example Output (Current Master Node - <Current_master_Node-FQDN>):
```
"cachedRoles": [
    "DATA",
    "UI"
],
"nodeIdentifier": "#####-####-####-####-####
```

Environment

VCF Operations 9.0.x
VCF Operations Fleet Management appliance 9.0.x

Cause

The master and replica node roles were swapped, but this change was not synchronized with the Fleet Management appliance. During a scheduled inventory sync, the VCF Operations CaSA API ( https://<Ops_Node_FQDN>/casa/cluster/status ) returned a cluster-status payload containing no MASTER node, as the actual master node reported itself as DATA. Fleet Management persisted this invalid state. Subsequent syncs failed immediately because VCF Ops host resolution requires a node typed MASTER.

Resolution

To resolve this issue, the missing roles must be updated and the stale environment must be cleared from Fleet Management:

Update the missing roles for the affected node by following the steps outlined in Broadcom KB 368959.
Restart the analytics service on the affected node by executing the following command: systemctl restart analytics.service
Verify that the service is running properly: systemctl status analytics.service
Confirm the CaSA cluster status now returns a valid MASTER node by querying the API endpoint: https://<Node FQDN>/casa/cluster/status
Once the CaSA endpoint returns the correct status, contact Broadcom Technical Support to clear the stale environment. Please reference this KB article (453809) in your case details.