"HTTP response code: 401 HTTP response body invalid credentials." during Deploy VCF Management Components in VCF 9.1

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• During the upgrade of the VCF Operations Management components from version 9.0 to 9.1, the Deploy VCF Management Components task fails with the error during the Import VCF Operations in Fleet Lifecycle subtask:

Message: Failed to configure LCM components
Remediation Message:
Reference Token: ######
Cause: Message: HTTP response code: 401 HTTP response body:("error":"access_denied","error_description":"invalid credentials. incorrect credentials. 2 attempts remaining"} HTTP response headers: {content-type=[application/json], date=[DD MMM YYYY HH:MM:SS], vary=[Accept-Encoding]}

Message: Failed to configure LCM components
Remediation Message:
Reference Token: ######
Cause: Message: HTTP response code: 401 HTTP response body: {"error":"access_denied","error_description":"invalid credentials. incorrect credentials. Account locked due to too many failed attempts"} HTTP response headers: {content-type=[application/json], date=[DD MMM YYYY HH:MM:SS], vary=[Accept-Encoding]}

• /var/log/vmware/vcf/domainmanager/domainmanager.log, on SDDC Manager : 

YYYY-MM_DDTHH:MM:SSZ DEBUG [vcf_dm,################################,####] [c.v.v.v.s.fleetlcm.FleetLcmService,dm-exec-107]  Creating FbsClient for <FLEET_LCM_FQDN> using VSP credentials
YYYY-MM_DDTHH:MM:SSZ ERROR [vcf_dm,################################,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-107]  [######] PUBLIC_LCM_COMPONENTS_CONFIGURE_FAILED Failed to configure LCM components
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to configure LCM components
        at com.vmware.vcf.vsp.services.fleetlcm.actions.ImportComponentFleetLcmAction.execute(ImportComponentFleetLcmAction.java:70)
        at com.vmware.vcf.vsp.services.fleetlcm.actions.ImportComponentFleetLcmAction.execute(ImportComponentFleetLcmAction.java:31)
Caused by: com.vmware.vsp.invoker.ApiException: Message:
HTTP response code: 401
HTTP response body: {"error":"access_denied","error_description":"invalid credentials. account is locked for 898 seconds"}
HTTP response headers: {content-type=[application/json], date=[DD MMM YY HH:MM:SS], vary=[Accept-Encoding]}

• The [email protected] account credentials expired/updated after the task started.

Environment

VCF Operations 9.1.x

Cause

This issue is caused by the expiry/rotation of the password for the [email protected] user for VCF Services runtime component during the task execution, this fails the API call to the VCF Services runtime to deploy the components.

Resolution

Contact Broadcom support for resolution : Creating Broadcom support