Unable to deploy VCF Automation from VCF Operations - Receiving error code LCMVCFA00013

Products

• VCF Automation
• VCF Operations

Issue/Introduction

• Attempting to deploy VCF Automation 9.0.x via Fleet Management fails at Stage 17
• VCF Automation fails with an error code "LCMVCFA00013 - Error occurred while getting refresh token using service account token"
• The /var/log/vrlcm/vmware_vrlcm.log on the Fleet Management appliance shows the following

INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.p.t.GetSddcManagerEndpointsFromVcfaTask] – Get SDDC Manager endpoints from VCF Automation task.
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.u.VcfaEndpointRestUtil] – Request to get refresh token using service account token.
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.VcfaRestClient] – Triggering request :: https://###.###.###/tm/oauth/provider/token
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.u.CustomTrustManager] – Certificate chain trusted
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.VcfaRestClient] – API Response got :: HttpResponseProxy

{HTTP/1.1 404 Not Found [vary: Accept-Encoding, date: Tue, DD MM YYYY HH:MM:SS GMT, connection: close, content-length: 0] [Content-Length: 0,Chunked: false]}
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.VcfaRestClient] – API Response Status : 404 Response Message :
ERROR vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.d.r.u.VcfaEndpointRestUtil] – Error occurred while getting refresh token using service account token.
ERROR vrlcm[1187] [pool-3-thread-77] [c.v.v.l.v.p.t.GetSddcManagerEndpointsFromVcfaTask] – Error occurred while getting SDDC Manager endpoints from VCF Automation account
INFO vrlcm[1187] [pool-3-thread-77] [c.v.v.l.p.a.s.Task] – Injecting task failure event. Error Code : 'LCMVCFA00013', Retry : 'true', Causing Properties : '

{ CAUSE :: }
'
INFO vrlcm[1187] [pool-3-thread-76] [c.v.v.l.v.p.u.VMSPServerRestUtil] – httpGetCallWithoutRetry url : /status?selector%3Dgroup+notin%28snapshots%2Cbackup%2Cnoncritical%29
ERROR vrlcm[1187] [pool-3-thread-76] [c.v.v.l.v.p.u.VMSPServerRestUtil] – Error while triggering API, kubeConfig YXYXYXYX passed YXYXYXYX auth token

• You have deleted and re-deployed the VCF Automation instance, and it fails with the same error code again

Environment

VCF Fleet Management 9.0.x

VCF Automation 9.0.x

Cause

The service account for integrations with VCF Operations is created on the Fleet management side. The tokens are created in the Tenant Manager database. During redeployment Fleet management recreates the integration accounts, but does not delete the old ones. Once the machine is deployed the new integration accounts do not work because their token is invalid and VCF Automation fails to deploy successfully. 

Resolution

This issue is currently being investigated by Broadcom Engineering.

 

Workaround:

1. Delete the newly deployed VCF Automation instance from the Fleet Manager components, ensuring you do not select the option to delete the appliances.
2. Go back to the Automation deployment and select Import.
3. Update the VIP IP, select the system and admin credentials, and proceed with the import operation.
4. The import task should complete successfully.