"Error connection Fleet management node, try again" when redeploying Fleet Manager appliance

Products

• VCF Operations

Issue/Introduction

• Previous Fleet Manager appliance was disconnected from VCF Operations using the Admin UI.
• A new Fleet Manager appliance was deployed but receive the error "Error connection Fleet management node, try again" when attempting to connect in the VCF Operations Admin UI.

Environment

VCF Operations 9.0.0.0

VCF Fleet Manager 9.0.0.0

Cause

The removal of the previous Fleet Manager was not successful.

Errors seen in /storage/vcops/log/casa/casa.log like

[com.vmware.vcops.casa.status.OperationCheckerService.verifyConsistentStateToAddNodes:1289] - Cluster is in inconsistent state : Node removal is in FAILED state

Resolution

1. Create cluster offline snapshots of all nodes in the VCF Operations cluster
   
   Snapshot Creation in VMware Aria Operations
2. Verify in the VMware Aria Operations Admin UI that the status of the cluster is Offline
3. Log in to all analytic nodes (primary, replica, data) in the VCF Operations cluster as root via SSH or vSphere console
4. Stop the Cluster and Slice Administration (CaSA) service on all analytic nodes
   

   systemctl stop vmware-casa

   Note: The Admin UI will become unavailable.

5. Clear the failure status from the casa.db.script file on all analytic nodes 
   

   sed -ri 's/"remove_node_state":"\w+"/"remove_node_state":"NONE"/g' /storage/db/casa/webapp/hsqldb/casa.db.script

6. Start the Cluster and Slice Administration (CaSA) service on all analytic nodes 
   

   systemctl start vmware-casa

   Note: It may take up to 5 minutes after starting the CaSA service for the Admin UI to become available.

7. Log in to the VCF Operations Admin UI and connect the new Fleet Manager appliance again
8. Bring the VCF Operations cluster back online.

Additional Information

VCF Operations unable to connect to VCF Fleet Management appliance with error, "Error connection Fleet management node, try again"