Aria Automation salt minion deployments fail with 500 server error

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Aria Automation deployments with saltstack minion resources fail with the following error:

Resource [/resources/compute/<COMPUTE_ID>]:: SaltStack resource with job id [<JOB_ID>] failed. Error:: Failed to get the current configuration status: Failed invoking [get_minion_deployments] RPC method for resource [minions]: Failed to authenticate: 500 Server Error: Internal Server Error for url: https://<FQDN>/account/login

In the RaaS logs you see failures similar to the following:

2024-06-06 16:24:51,935 [var.lib.raas.unpack._MEIPSmyD1.raas.mods.rest.account             ][WARNING :216 ][Webserver:347] Authentication lock is in use by another raas instance, retrying 9/10.
2024-06-06 16:24:52,362 [asyncio                                                           ][DEBUG   :54  ][ForkPoolWorker-3:388] Using selector: EpollSelector
2024-06-06 16:24:52,940 [pals.core                                                         ][DEBUG   :216 ][Webserver:347] Lock result was: False
2024-06-06 16:24:52,940 [var.lib.raas.unpack._MEIPSmyD1.raas.mods.rest.account             ][ERROR   :216 ][Webserver:347] Unable to acquire aunthentication lock. Login from <USER> rejected. Error: Lock acquire failed for "<USER>". result was: {retval}.
2024-06-06 16:24:52,940 [tornado.application                                               ][ERROR   :1871][Webserver:347] Uncaught exception POST /account/login (<IP_ADDR>)
HTTPServerRequest(protocol='https', host='<HOSTNAME>', method='POST', uri='/account/login', version='HTTP/1.1', remote_ip='<IP_ADDR>')
Traceback (most recent call last):
  File "raas/utils/org.py", line 184, in wraps
    return func(*args, **kwargs)
  File "/var/lib/raas/unpack/_MEIPSmyD1/raas/mods/rest/account.py", line 173, in post
    with locker.lock(f"user:{username}", blocking=False):
  File "pals/core.py", line 154, in __enter__
    self._acquire()
  File "pals/core.py", line 127, in _acquire
    raise AcquireFailure(self.name, 'result was: {retval}')
pals.core.AcquireFailure: Lock acquire failed for "<USER>". result was: {retval}.

Cause

The user account leveraged by Aria Automation to integrate with Aria Automation Config is an LDAP user

The frequent logins from Automation to the Config RPC API is causing issues with account locking

Resolution

Create a local user account in Aria Automation Config that can be used for the integration to reduce the calls from Config to the LDAP provider