VCF Operations upgrade fails with error LCMVCFA00007: Unable to get access token from VCF Automation

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When attempting to upgrade VCF Operations from version 9.0.0 to 9.0.1, the upgrade fails during the inventory sync stage. The following error is displayed in the UI:LCMVCFA00007: Unable to get access token from VCF Automation
  
• In the Fleet Manager appliance logs located at /var/log/vrlcm/vmware_vrlcm.log, you may see entries indicating a 503 Service Unavailable error when trying to reach the VCF Automation endpoint.
  Log snippets:

  YYYY-MM-DDTHH:MM:SS. INFO vrlcm[#######] [pool-3-thread-61] [c.v.v.l.v.d.r.VcfaRestClient] – API Response Status : 503 Response Message : no healthy upstream
  YYYY-MM-DDTHH:MM:SS INFO vrlcm[#######] [pool-3-thread-61] [c.v.v.l.v.d.r.VcfaRestClient] – Response of get token API call :: { "statusCode" : 503, "responseMessage" : "no healthy upstream" ...

 

Environment

VCF Operations 9.x

Cause

This issue occurs because the services (Pods) on the VCF Automation appliance are not in a healthy state. When the upgrade process attempts to sync inventory, it cannot retrieve the necessary access token from the Automation appliance due to the 503 Service Unavailable status.

Resolution

To resolve this issue, you must ensure the VCF Automation services are running correctly before retrying the upgrade:
1. Log in to the VCF Automation appliance as root via SSH as vmware-system-user and then switch to root by executing following command:

su -u

2. Verify the status of the pods by running the following commands:

kubectl get pods -n prelude
kubectl get pods -A

3. If any pods are in a CrashLoopBackOff, Error, or Pending state, troubleshoot the Automation appliance to restore service health. You may open a Broadcom Support Case with VCF Automation team for the same.
4. Once all VCF Automation services are confirmed to be healthy and the UI is accessible, return to the upgrade interface and Retry the inventory sync or upgrade task.