"VCF Operations Fleet Management Is Not Ready" from VCF OPS UI

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• VCF Operations Fleet Management status is not ready.
• Admin UI shows Fleet Management status as Connected.
• /storage/log/vcops/log/web-vcf.log in VCF operations shows:
  
  versionCheck API request was unsuccessful for ops-lcm, code: 403.

YYYY-MM-DDTHH:MM:SS.606Z ERROR web-vcf 1223 [ops@4413 threadId="4582" threadName="Thread-18"] [com.vmware.vcf.operations.ui.util.ESPluginUpdater.error] - ESPluginUpdater: versionCheck API request was unsuccessful for ops-lcm, code: 403.
  
  
• Remediation of Fleet Manager admin and root password fails.  Red banner shows the following: 
  
  Remediate for appliance <fleetmanagerfqdn> has failed
  
  
• HAR capture when navigating to Fleet Management → Lifecycle →  VCF Management shows the below error.  You can use the following a guide to capturing HAR How to collect a HTTP Archive (HAR) log file for troubleshooting Cloud Director or VCF Business Service Console\  
  
  GET https://<VCFOpsFQDN>/vcf-operations/plug/ops-lcm/lcm/bootstrap/api/status 403 (Forbidden)

Environment

VCF 9.0.x

Cause

One of the VCF Operations cluster nodes root password was changed by command line which is not recommended.As a result VCF Operations is out of sync with Fleet Management.

Resolution

1. Change the root password of all VCF Operations nodes to the same (by using SSH passwd command).
2. Triggered Inventory Sync of VCF Operations from Fleet Management.
3. Reboot Fleet Management

Additional Information

For more information on managing passwords in VCF 9.0 refer to document Managing Passwords for VMware Cloud Foundation Components

NOTE: There could be multiple potential causes for this issue, the current solution may not apply to all scenarios given the variety of potential root causes.