VCF Identity Broker deployment from VCF Fleet Management UI is hung fetching the infrastructure details

Products

VCF Operations

Issue/Introduction

VCF Fleet Management UI froze during the 'identity-broker' deployment at the 'fetching infrastructure details' step.

The "/var/log/vrlcm/vmware_vrlcm.log" reports below error:

INFO vrlcm[1254] [http-nio-8080-exec-8] [c.v.v.l.c.a.InternalOnlyApiAspect]  -- Internal Only Check for: execution(ResponseEntity com.vmware.vrealize.lcm.locker.controller.CredentialController.getPassword(String))
INFO vrlcm[1254] [http-nio-8080-exec-8] [c.v.v.l.u.v.VropsUtil]  -- Getting all vidb identity providers for vrops.
INFO vrlcm[1254] [http-nio-8080-exec-8] [c.v.v.l.u.LemansRestClient]  -- Url :: https://VCFOps-FQDN/suite-api/api/auth/token/acquire
ERROR vrlcm[1254] [http-nio-8080-exec-8] [c.v.v.l.u.LemansRestClient]  -- {"type":"Error","message":"The provided username/password YXYXYXYX is not valid. Please try again.","httpStatusCode":401,"apiErrorCode":401}
ERROR vrlcm[1254] [http-nio-8080-exec-8] [c.v.v.l.u.LemansRestClient]  -- <401 UNAUTHORIZED Unauthorized,{"type":"Error","message":"The provided username/password YXYXYXYX is not valid. Please try again.","httpStatusCode":401,"apiErrorCode":401},[]>
ERROR vrlcm[1254] [http-nio-8080-exec-8] [c.v.v.l.l.c.DataCenterController]  -- Exception while getting all vCenters : com.vmware.vrealize.lcm.common.exception.userinput.vrops.VropsVidbInstanceException: Error occurred while getting identity-providers from VCF Operations. Unable to fetch Suite API token
com.vmware.vrealize.lcm.common.exception.userinput.vrops.VropsVidbInstanceException: Error occurred while getting identity-providers from VCF Operations. Unable to fetch Suite API token
        at com.vmware.vrealize.lcm.lcops.service.DataCenterServiceImpl.getAllVcfInstancesOfVidb(DataCenterServiceImpl.java:1511) ~[vmlcm-lcopsservice-core-9.0.0.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.lcops.service.DataCenterServiceImpl.getAllVCentersV3(DataCenterServiceImpl.java:1294) ~[vmlcm-lcopsservice-core-9.0.0.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.lcops.service.DataCenterServiceImpl$$FastClassBySpringCGLIB$$b3f2a631.invoke(<generated>) ~[vmlcm-lcopsservice-core-9.0.0.0-SNAPSHOT.jar!/:?]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.42.jar!/:5.3.42]

Also "/storage/log/vcops/log/analytics.audit-##########.log" reports VCF Operations "admin" account getting locked.

INFO  audit 5821 [ops@4413 threadId="959" userId="###########" username="admin" authSource="LOCAL" session="n/a" clientIP="Source-IPAddress" auditID="AUTHENTICATION_LOGIN"] [AuditLog.log] - Log in failed. Origin: "REST_API". Due to the following error: "Invalid credentials".
INFO  audit 5821 [ops@4413 threadId="959" userId="###########" username="admin" authSource="LOCAL" session="n/a" clientIP="Source-IPAddress" auditID="AUTHENTICATION_LOGIN"] [AuditLog.log] - Log in failed. Origin: "REST_API". Due to the following error: "Account is locked".

The "/storage/log/vcops/logs/casa/casa-audit.log" will report the "admin" account password update as given below:

INFO  casa.audit 1587 [ops@ ##### threadId=" #####" threadName="ajp-nio-##.##.#.#-8011-exec-17" requestId="ui000YDK"] [casaAuditLogger.log:78] - UserId : casa, UserName : casa, AuthSource : unknown, Session : ###################, Category : UPDATE_PASSWORD - User updating system admin password

Environment

VCF Operations 9.x
VCF Fleet Management 9.x

Cause

This issue occurred as the "admin" password for VCF Operations was updated manually via the GUI, bypassing Unified Password Management (UPM).
Due to which the VCF Fleet Locker was not automatically synchronized and the "admin" account became disconnected under VCF Operations UI >> Fleet Management >> Passwords >> VCF Management

Resolution

Follow the below steps to resolve this issue and proceed with VCF Identity Broker deployment from VCF Fleet Management UI

Remediate the VCF Operations "admin" password using the UPM UI to sync with the Fleet Management Locker
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/fleet-management/manage-passwords/remediate-passwords.html
Confirm the VCF Operations "admin" account is showing "Active" under VCF Operations UI >> Fleet Management >> Passwords >> VCF Management.
Re-initiate the VCF Identity Broker deployment from VCF Fleet Management UI.

Additional Information

As a part of vIDB deployment infrastructure selection, Fleet Management validate any external vIDB is deployed in any of the vCenter or not using the VCF Operations API.
Fleet Management maintains the 'admin' password in the Locker which is provided as part of the Initial Registration process. Same Locker password is used to access any VCF Operations API.
If any Password change is happened via the UPM (Unified Password Management), it will automatically sync with Fleet Management Locker.
If the Password is changed outside the UPM like directly using GUI, it won't get synced with Locker automatically and while authenticating the VCF Ops API from Fleet management it throws 401 Authentication error due to the password mismatch.