"Failed to get admin token: Tenant does not exist." error in Aria Suite Lifecyle Onboard tenant API when original primary Identity Manager node is changed

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• VMware Identity manager is deployed in a cluster. Attempts to leverage the Aria Suite Lifecyle onboard api call fail if the primary Identity manager node has been changed since initial deployment.
• The /var/log/vrlcm/vmware_vrlcm.log file contains an error similar to:

2022-10-20 10:12:20.103 INFO [pool-3-thread-33] c.v.v.l.v.c.t.t.VidmOnboardTenantTask - -- Starting :: vIDM Add tenant Task
2022-10-20 10:12:20.108 INFO [scheduling-1] c.v.v.l.c.u.EventExecutionTelemetryUtil - -- Stop Instrumenting EventMetadata.
2022-10-20 10:12:20.152 ERROR [pool-3-thread-33] c.v.v.l.v.c.t.t.VidmOnboardTenantTask - -- Failed to login to vIDM vIDMServer{host=<FQDN>, tenant=null}.
com.vmware.vrealize.lcm.common.exception.LcmException: Failed to get admin token: Tenant does not exist.
        at com.vmware.vrealize.lcm.vidm.driver.rest.util.VidmServerRestUtil.loginLocalAdmin(VidmServerRestUtil.java:237) ~[vmlcm-vidmplugin-driver-8.10.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.vidm.core.task.tenant.VidmOnboardTenantTask.execute(VidmOnboardTenantTask.java:129) [vmlcm-vidmplugin-core-8.10.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:63) [vmlcm-engineservice-core-8.10.0-SNAPSHOT.jar!/:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2022-10-20 10:12:20.154 INFO [pool-3-thread-33] c.v.v.l.p.a.s.Task - -- Injecting task failure event. Error Code : 'LCMVIDM70000', Retry : 'true', Causing Properties : '{ CAUSE :: hostName === vidmTenant === vidmAdminUser === vidmAdminPassword YXYXYXYX }'
com.vmware.vrealize.lcm.common.exception.LcmException: Failed to get admin token: Tenant does not exist.
        at com.vmware.vrealize.lcm.vidm.driver.rest.util.VidmServerRestUtil.loginLocalAdmin(VidmServerRestUtil.java:237) ~[vmlcm-vidmplugin-driver-8.10.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.vidm.core.task.tenant.VidmOnboardTenantTask.execute(VidmOnboardTenantTask.java:129) [vmlcm-vidmplugin-core-8.10.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:63) [vmlcm-engineservice-core-8.10.0-SNAPSHOT.jar!/:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

Environment

VMware vRealize Suite Lifecycle Manager 8.x

Resolution

A resolution for the issue is scheduled to be included in the Aria Suite Lifecycle 8.12 release

Workaround:

To workaround the issue:

 

1. Find the current Primary vIDM node.
   a. cat /usr/local/etc/pgpool.pwd
   b. su root -c "echo -e 'password'|/opt/vmware/vpostgres/current/bin/psql -h localhost -p 9999 -U pgpool postgres -c
     \"show pool_nodes\""
   c. Update the 'password' in b with output from a.
2. Failback vPostgres to Node1 (Node 1 during vRLCM deployment)
        a. Reboot/Shutdown the current Primary Identity Manager Node (Node 2 or Node 3) to failover automatically to Node1.

Additional Information

Impact/Risks:
The issue only affects the Aria Suite Lifecyle onboarding API. Requests to onboard tenants using the UI are unaffected by this issue.