"VCF Operations Fleet Management Is Not Ready" from VCF OPS UI

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When browsing Fleet Management Lifecycle in VMware Cloud Foundation (VCF) Operations UI you see the error:
  
  

  "VCF Operations Fleet Management is not Ready"

• Admin UI shows Fleet Management status as Connected.
• In /storage/log/vcops/log/web-vcf.log on VCF Operations appliance you see the below entries:
  
  YYYY-MM-DDTHH:MM:SS.606Z ERROR web-vcf 1223 [ops@4413 threadId="4582" threadName="Thread-18"] [com.vmware.vcf.operations.ui.util.ESPluginUpdater.error] - ESPluginUpdater: versionCheck API request was unsuccessful for ops-lcm, code: 403.
  
  
• Remediation of Fleet Manager admin and root password fails.  Red banner shows the following: 
  
  Remediate for appliance <fleetmanagerfqdn> has failed
  
  
• HAR capture when navigating to Fleet Management → Lifecycle →  VCF Management shows the below error.  You can use the following a guide to capturing HAR How to collect a HTTP Archive (HAR) log file for troubleshooting Cloud Director or VCF Business Service Console\  
  
  GET https://<VCFOpsFQDN>/vcf-operations/plug/ops-lcm/lcm/bootstrap/api/status 403 (Forbidden)

Environment

VMware Cloud Foundation 9.0.x

Cause

One of the VCF Operations cluster nodes root password was changed by command line which is not recommended. As a result VCF Operations is out of sync with Fleet Management.

Resolution

1. Change the root password of all VCF Operations nodes to the same (by using SSH passwd command).
   If the root password is not working and you are not able to SSH the one or all the nodes follow the KB How to Reset the Root Password for VCF Operations
2. Triggered Inventory Sync of VCF Operations from Fleet Management:
   
   2.a. Log in to the VCF Operations console.
   2.b. Navigate to Fleet Management > Lifecycle > VCF Management > Components.
   2.c. Select an existing integrated component (e.g., VCF Operations) and click the TRIGGER INVENTORY SYNC button.
   2.d. Wait for the task to complete. This forces the Fleet Management Appliance to fetch the latest vCenter and SDDC Manager configurations and update its internal database.
   2.e. Retry the Add Component workflow for VCF Operations for Networks. The updated cluster name will now be available in the infrastructure configuration drop-down.
   
   
3. Reboot Fleet Management
4. If the issue still persisted, take snapshots for VCF Operations and Fleet Management as per below KB:
   
   How to create a snapshot for VCF Operations Fleet Manager appliance
   Create a Snapshot Before You Upgrade VCF Operations
   
   
5. Open Fleet Management UI by entering https://<fleetmanagementfqdn> in the browser.
6. Navigate to Lifecycle → Components → Locate the VCF Operations → Click on 3 dots → Click on Delete → Ensure you do not select to delete the VM from vCenter.
7. If the Delete option on the Fleet Management UI is not available follow the steps in the KB Troubleshooting VCF Operations fleet management node Integration Failures in the VCF Operations
8. Reconnect Fleet Management on VCF Operations Admin UI as per documentation at Register the VCF Operations fleet management Appliance with VCF Operations

Additional Information

For more information on managing passwords in VCF 9.0 refer to document Managing Passwords for VMware Cloud Foundation Components

NOTE: There could be multiple potential causes for this issue, the current solution may not apply to all scenarios given the variety of potential root causes.