Unable to deploy VCF Automation from VCF Operations - Receiving error code LCMVCFA00013

Products

• VCF Automation
• VCF Operations

Issue/Introduction

• Attempting to deploy VCF Automation 9.0.x via Fleet Management fails at Stage 17
• VCF Automation fails with an error code "LCMVCFA00013 - Error occurred while getting refresh token using service account token"
• The /var/log/vrlcm/vmware_vrlcm.log on the Fleet Management appliance shows the following

INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.p.t.GetSddcManagerEndpointsFromVcfaTask] – Get SDDC Manager endpoints from VCF Automation task.
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.u.VcfaEndpointRestUtil] – Request to get refresh token using service account token.
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.VcfaRestClient] – Triggering request :: https://###.###.###/tm/oauth/provider/token
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.u.CustomTrustManager] – Certificate chain trusted
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.VcfaRestClient] – API Response got :: HttpResponseProxy

{HTTP/1.1 404 Not Found [vary: Accept-Encoding, date: Tue, DD MM YYYY HH:MM:SS GMT, connection: close, content-length: 0] [Content-Length: 0,Chunked: false]}
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.VcfaRestClient] – API Response Status : 404 Response Message :
ERROR vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.u.VcfaEndpointRestUtil] – Error occurred while getting refresh token using service account token.
ERROR vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.p.t.GetSddcManagerEndpointsFromVcfaTask] – Error occurred while getting SDDC Manager endpoints from VCF Automation account
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.p.a.s.Task] – Injecting task failure event. Error Code : 'LCMVCFA00013', Retry : 'true', Causing Properties : '

{ CAUSE :: }
'
INFO vrlcm[1187] [pool-3-thread-76] [c.v.v.l.v.p.u.VMSPServerRestUtil] – httpGetCallWithoutRetry url : /status?selector%3Dgroup+notin%28snapshots%2Cbackup%2Cnoncritical%29
ERROR vrlcm[1187] [pool-3-thread-76] [c.v.v.l.v.p.u.VMSPServerRestUtil] – Error while triggering API, kubeConfig YXYXYXYX passed YXYXYXYX auth token

• You have deleted and re-deployed the VCF Automation instance, and it fails with the same error code again

Similarly, VCF Automation objects may appear as down or fail to authenticate in VCF Operations after performing a VCF Automation backup and restore (especially in cross-cluster scenarios), failing with the same "LCMVCFA00013 - Error occurred while getting refresh token" error.

Environment

VCF Fleet Management 9.0.x

VCF Automation 9.0.x

Cause

The service account for integrations with VCF Operations is created on the Fleet management side. The tokens are created in the Tenant Manager database. During redeployment Fleet management recreates the integration accounts, but does not delete the old ones. Once the machine is deployed the new integration accounts do not work because their token is invalid and VCF Automation fails to deploy successfully. 

 

Similarly, when restoring a VCF Automation backup (particularly to a different cluster), the existing refresh tokens and service accounts become mismatched between VCF Operations and VCF Automation, rendering the tokens invalid.

Resolution

Note: This issue is a known architectural limitation. Performing a backup and restore of a VCF Automation cluster, especially in a cross-cluster scenario, is currently an unsupported workflow. Broadcom Engineering is aware of this behavior; however, there are no plans to implement a code fix for this authentication mismatch as it arises from a non-recommended workflow.

To resolve this issue in either the redeployment or restore scenario, please perform the workaround below.

 

Workaround:

1. Delete the newly deployed VCF Automation instance from the Fleet Manager components, ensuring you do not select the option to delete the appliances.
2. Go back to the Automation deployment and select Import.
3. Update the VIP IP, select the system and admin credentials, and proceed with the import operation.
4. The import task should complete successfully.

Additional Information

Note: This issue is a known architectural limitation. Performing a backup and restore of a VCF Automation cluster, especially in a cross-cluster scenario, is currently an unsupported workflow.

Broadcom Engineering is aware of this behavior; however, there are no plans to implement a code fix for this authentication mismatch as it arises from a non-recommended workflow. Users must perform the manual re-import procedure described in the Workaround section to restore connectivity after a restore operation.