VMware Cloud Services Gateway's local user account gets locked for vSphere+

Products

• VMware Cloud on AWS

Issue/Introduction

Symptoms:

VMware Cloud Services Gateway due to various exceptions and failure scenarios leads to a critical local OS account with username 'aap-super-user' password to get locked thus rendering the application unable to rotate the password automatically or serve API requests that is dependent on this user's account.

Log snippets: (Reference purpose only)

[2023-08-23 00:00:00:040 GMT] [identity-agent] Failed to update password for aap-super-user org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal expired.
Failed to fetch session token 500 Internal Server Error: "{"messages":[{"id":"vapi.security.authentication.exception","default_message":"Exception in involving authentication handler User account locked.

[2023-08-23 00:00:00:040 GMT] org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "["messages":[["id":"vapi.security.authentication.exception","default_message":"Exception in invoking authentication handler User password expired.","args":["User password expired."]]],"error_type":"INTERNAL_SERVER_ERROR"]"

[2023-08-23 00:00:00:040 GMT] org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized: "["messages":[["id":"vapi.security.authentication.invalid","default_message":"Unable to authenticate user","args":[]]],"error_type":"UNAUTHENTICATED"]"

Environment

VMware vSphere+ Cloud Services

Cause

Different failure scenarios not limited to root password expiry and not being able to rotate the root credentials timely can lead to this situation.

Resolution

This is a proactive step required to be performed to address Cloud Gateway upgrade issues.

Workaround:

Steps to be followed:

1. Download the two files attached to this article
2. SCP the downloaded files to the Gateway. Ex - scp unlock_aap_super_user* root@<GW_IP>:/root/
3. SSH into the Gateway
4. Run the below commands
   • chmod 755 unlock_aap_super_user.sh
   • chmod 755 unlock_aap_super_user.py
   • sh unlock_aap_super_user.sh
      

Additional Information

Impact/Risks:

VMware will not be able to perform any VMware Cloud Services Gateway upgrade leading to loss of some vSphere+ functionality.