Error: "VCF Operations Fleet Management is Not Ready" when navigating to VCF Management in VCF Operations 9.x

Products

• VCF Operations

Issue/Introduction

1. The VMware Cloud Foundation (VCF) Operations Fleet Management state displays as "VCF Operations Fleet Management is Not Ready". Attempts to disconnect or reconnect the node via the user interface (UI) failing with the following error message:
   

   Error connection Fleet Management node, try again

2. When investigating the logs on the Fleet Management and VCF Operations cluster, a 'No route to host (Host unreachable)' error is observed when the system attempts to reach the VMware Cloud Foundation Automation (VCFA) instance VIP.
   

   ####-##-##T10:19:05.620Z ERROR vrlcm[1239] [pool-3-thread-38] [c.v.v.l.v.p.t.VcfaDeleteCapabilitiesTask]  -- Error while deleting capabilities from VCFA
   com.vmware.vrealize.lcm.vcfa.common.exception.UnableToGetVcfaTokenException: Could not get VCF Automation token for host: example.com username: admin@System. No route to host (Host unreachable)
   Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)

3. Navigating to VCF Operations UI > Fleet Management >  Lifecycle > VCF Management > Component section shows that the 'VCF Component' is in disconnected state

Environment

VCF Operations 9.0.x 

Cause

• The VCF Operations environment cannot be removed or reconnected because the Fleet Management appliance is blocked by a routing failure to the VCFA instance VIP.
• This network inaccessibility prevents the completion of API requests, causing deletion and reconnection tasks to hang or fail.

Resolution

Follow the steps below to resolve the issue

NOTE: Before performing the following procedure, ensure that you take snapshots of  VCF Operations nodes and the Fleet Management appliance.

Removing VCF Component:

Method 1: WEB UI

Remove the inaccessible components from the Fleet Manager UI.

1. Navigate to Fleet Management > Lifecycle > Components > Automation > Delete.
   • NOTE: There is a risk of deleting the component if the deleteFromVcenter value is checked.

Method 2: API

1. 1. Refer to How to Authorize VCF Operations Fleet Management API to access and authorize with the Fleet Management API.
   2. Open the Swagger UI for Fleet Management by navigating to VCF Operations > Developer Center > APIs & SDKs > Fleet Management API.
   3. Follow the instructions detailed in "Error connection Fleet management node, try again" while connecting VCF Fleet manager to the VCF Operations.
   4. In the Fleet Management API, under the Environment Controller section, select the Delete Environment option for the VCF Operations instance.
   5. Ensure the deleteFromVcenter value is set to false.
   6. Submit the request.

Re-add the Fleet Manager:

1. 1. Open a browser and access the VCF Operations Admin UI at 'https://<vcf-operations-fqdn>/admin'.
   2. Log in as the local user 'Admin'.
   3. Locate the Fleet Management tile in the middle of the screen and click Connect.
   4. Enter the Fleet Manager IP or FQDN and the admin password configured during OVF/OVA deployment.
   5. Click Test Connection, accept any SSL certificate warnings, and click Next.
   6. Provide the admin credentials for your primary VCF Operations node and click Finish.
   7. Wait a few minutes until the status indicator updates to Connected

Additional Information

Reference Link: Re-add the Fleet Manager via the VCF Operations Admin UI