"VCF Operations Fleet Management Is Not Ready" in VCF Operations UI

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

1. Browsing Fleet manager passwords in VCF Operations UI shows "Credentials as disconnected".
2. When browsing Fleet Management Lifecycle in VMware Cloud Foundation (VCF) Operations UI you see the error:

   "VCF Operations Fleet Management is not Ready"

3. Admin UI shows Fleet Management status as Connected.
4. In /storage/log/vcops/log/web-vcf.log on VCF Operations appliance you see the below entries:

   YYYY-MM-DDTHH:MM:SS.606Z ERROR web-vcf 1223 [ops@4413 threadId="4582" threadName="Thread-18"] [com.vmware.vcf.operations.ui.util.ESPluginUpdater.error] - ESPluginUpdater: versionCheck API request was unsuccessful for ops-lcm, code: 403.

5. Remediation of Fleet Manager admin and root password fails.  Red banner shows the following: 

   Remediate for appliance <fleetmanagerfqdn> has failed

6. HAR capture when navigating to "Fleet Management → Lifecycle →  VCF Management" shows the below error.

   GET https://<VCFOpsFQDN>/vcf-operations/plug/ops-lcm/lcm/bootstrap/api/status 403 (Forbidden)

Environment

VCF Operations 9.0.x

Cause

One of the VCF Operations cluster nodes root password was changed by command line which is not recommended. As a result VCF Operations is out of sync with Fleet Management.

Resolution

1. Change the root password of all VCF Operations nodes to the same (by using SSH passwd command).
   If the root password is not working and you are not able to SSH the one or all the nodes follow the KB How to Reset the Root Password for VCF Operations.
2. Triggered Inventory Sync of VCF Operations from Fleet Management:
   1. Log in to the VCF Operations console.
   2. Navigate to Fleet Management > Lifecycle > VCF Management > Components.
   3. Select an existing integrated component (e.g., VCF Operations) and click the TRIGGER INVENTORY SYNC button.
   4. Wait for the task to complete. This forces the Fleet Management Appliance to fetch the latest vCenter and SDDC Manager configurations and update its internal database.
   5. Retry the Add Component workflow for VCF Operations for Networks. The updated cluster name will now be available in the infrastructure configuration drop-down.
3. Reboot Fleet Management
4. If the issue still persisted, take snapshots for VCF Operations and Fleet Management as per below KB:
   1. How to create a snapshot for VCF Operations Fleet Manager appliance
   2. Create a Snapshot Before You Upgrade VCF Operations
5. Open Fleet Management UI by entering https://<fleetmanagementfqdn> in the browser.
6. Navigate to Lifecycle → Components → Locate the VCF Operations → Click on 3 dots → Click on Delete
   Note: Ensure you do not select to delete the VM from vCenter.
7. If the Delete option on the Fleet Management UI is not available follow the steps in the KB Troubleshooting VCF Operations fleet management node Integration Failures in the VCF Operations
8. Reconnect Fleet Management on VCF Operations Admin UI as per documentation at Register the VCF Operations fleet management Appliance with VCF Operations

Additional Information

1. For more information on managing passwords in VCF 9.0 refer to document Managing Passwords for VMware Cloud Foundation Components
   • Note: There could be multiple potential causes for this issue, the current solution may not apply to all scenarios given the variety of potential root causes.
2. This issue can also occur if the Fleet Management appliance VM root password is expired.
3. Reset of the root password of the Fleet Management appliance resolved the issue - VCF Operations Fleet Management Is Not Ready from VCF OPS UI
4. Guide to capture HAR logs: How to collect a HTTP Archive (HAR) log file for troubleshooting