vRealize/Aria Operations admin account getting locked even after password reset

Products

• VMware Aria Suite

Issue/Introduction

 

        VMware Aria Operations (Formerly known as vRealize Operations Manager) admin account shows an Incorrect User name/Password even after providing the correct credentials.

        After resetting the password using How to Reset the Admin Password in VMware Aria Operations, the issue reappears after a few successful logins.

 

 

• You can check the number of failed_attempts in /storage/vcops/user/conf/adminuser.properties, as per the example below:

root@<hostname> [ ~ ]# cat /storage/vcops/user/conf/adminuser.properties
#Properties for vCOps Admin user
#Tue Feb 20 11:30:09 UTC 2024
hashed_password=<Password HASH removed>
failed_attempts=5675
time_last_failure=1708428609792
username=admin

Environment

VMware vRealize Operations 8.x

Aria Operations 8.x

Cause

• This issue occurs when a user changes the VMware Aria Operations (Formerly known as vRealize Operations Manager) admin account password.
• The new password is not automatically updated in the endpoints that are linked to VMware Aria Operations, for example, Skyline Collector, Aria Operations for Logs (formerly vRealize Log Insight), etc.
• These endpoints make several API connect attempts to VMware Aria Operations (Formerly known as vRealize Operations Manager) using the old password. Since the password has been changed, the admin account keeps getting locked.
• An integrated Management Pack may be configured to use the admin account and this password will not be updated automatically.
  • The analytics.audit.log may contain an error similar to this if one of the integrated management packs (adapters) is causing the lock.
    2024-09-04T11:37:34,812+0000 - UserId : XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX, UserName : admin, AuthSource : LOCAL, Session : n/a, ClientIP : 127.0.0.1, Category : AUTHENTICATION_LOGIN - Log in failed. Origin: "REST_API". Due to the following error: "Invalid credentials".

Resolution

Steps to resolve the issue:

1. SSH to the primary node of Aria Operations. (If using a load balancer execute the commands on all the analytics nodes)

2. The following commands can be used to help identify source of account lockout, command 3 is likely the most useful in most cases, even though it does not distinguish between admin or other accounts:

Command 1:
grep -a /auth\/token\/acquire.*\"username\":\"admin\" /storage/log/vcops/log/http_api.log*

Example from lab server:

2024-02-20T11:52:17,086+0000 INFO [ajp-nio-127.0.0.1-8010-exec-3, 71EifC1H7nCAYsIGrOmdlRES9jEhPzfb] utils.HttpRequestLogger - After request [POST /suite-api/api/auth/token/acquire?_no_links=true, client=ip_address, payload={"username":"admin","password":"*****"}]
2024-02-20T11:52:17,202+0000 INFO [ajp-nio-127.0.0.1-8010-exec-10, ivHLQF7tmorqYaAbt0J9T98EoWriqla7] utils.HttpRequestLogger - After request [POST /suite-api/api/auth/token/acquire?_no_links=true, client=ip_address, payload={"username":"admin","password":"*****"}]
2024-02-20T11:52:17,359+0000 INFO [ajp-nio-127.0.0.1-8010-exec-2, 63ITvAa4VvTVephLHe4J0ABuFlx0VnoK] utils.HttpRequestLogger - After request [POST /suite-api/api/auth/token/acquire?_no_links=true, client=ip_address, payload={"username":"admin","password":"*****"}]

Notes:

• The above command will search all http_api.log* files for failed logins using admin account, but will give no indication as to whether these logins were successful or unsuccessful.
• To search the latest log file only, remove the * at the end of the log filename in the command

Command 2:
grep -a /auth\/token\/acquire.*\"username\":\"admin\" /storage/log/vcops/log/http_api.log | awk '{print $11}' | uniq -c

Example from lab server:

root@<hostname> [ ~ ]# grep -a /auth\/token\/acquire.*\"username\":\"admin\" /storage/log/vcops/log/http_api.log | awk '{print $11}' | uniq -c
      1 client=ip_address,
      5 client=ip_address,
      2 client=ip_address,
      1 client=ip_address,
      1 client=ip_address,
      1 client=ip_address,
   6636 client=ip_address_1,
     16 client=ip_address_2,
    109 client=ip_address_3,

Notes:

• The above command will search the same log file as in command 1, and count how many logins there were per IP address. This is mostly useful to determine where admin logins are coming from and how many there were coming in from the same source.
• Repeating this command after a few seconds shows which endpoint is repeatedly attempting to log in.
• ip_address_1, ip_address_2, and ip_address_3 in output above is from a 3 node Operations for Logs (LogInsight) cluster that was used to lock out admin account in our test environment.

Command 3:
grep -a /auth\/token\/acquire.* /storage/log/vcops/log/suite-api/localhost_access_log.txt | awk '{print $3}' | sort | uniq -c

Example from lab server:

root@<hostname> [ ~ ]# grep -a /auth\/token\/acquire.* /storage/log/vcops/log/suite-api/localhost_access_log.txt | awk '{print $3}' | sort | uniq -c
  10839 127.0.0.1
      1 ip_address
     16 ip_address
      7 ip_address
   6784 ip_address

Notes:

• The above command will search /storage/log/vcops/log/suite-api/localhost_access_log.txt log file.
• Log file reports the HTTP codes (200/401/etc) as well as IP address of endpoint.
• As with command 2 above, repeat after a few seconds to determine IP where the count increases.
• This command does not distinguish between admin account or other accounts

Example output from /storage/log/vcops/log/suite-api/localhost_access_log.txt:

Successful login:
[20/Feb/2024:12:17:28 +0000] ip_address- - "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1" 200 204 19

Failed login:
[20/Feb/2024:11:29:31 +0000] ip_address- - "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1" 401 125 27

Note the 200 HTTP code and the 401 HTTP code in the example log output above.
 

Summary:

• The purpose of the above commands is to help identify which remote system is attempting to log in, and is the likely cause of the account lockout.
• Note that even though an endpoint is unable to log in, resulting in an increase in failed login attempts, this may not be the cause of the initial lockout. However, in most cases we see that it's the integration with endpoints that are failing to log in that is the cause of the account lockout, after Admin password was changed on Aria Operations.
• If Aria Operations is behind a load balancer, and endpoint use VIP to access Aria Operations, the IP shown will normally refer to the LB itself. You will need to investigate the LB to identify the source.
• After verifying that all management packs (adapters) are not configured to use the Aria Operations admin account you can ignore logins from 127.0.0.1, other analytics nodes, and Cloud Proxies/Remote Collectors that have joined the cluster. We are normally only interested in addresses that are not members of the cluster. Command 2 and command 3 above are the most useful to determine where login attempts are coming from, and repeating the commands usually show an increased count from when you repeat the command.

3. After you've identified the source of the lockouts, and changed the integration password on the endpoint, follow KB Log in to Admin and Product UI as admin fails in vRealize/Aria Operations to reset the lockout on the admin account.

 

Additional Information

For information on resetting the Admin password, see How to reset the admin password in vRealize Operations (326391).
For information on resetting the Admin account lockout, see Log in to Admin and Product UI as admin fails in vRealize/Aria Operations (326396) .

Impact/Risks:
Due to this issue, the admin user is not able to access Aria Operations Product and Admin UI

This article does not apply to other products with Aria Operations in the name such as:

• Aria Operations for Logs
• Aria Operations for Networks.