Configuration back up scheduler failing with error "Lock acquisition timeout" or GUI/CLI inaccessible HTTP 500 response

Products

• VMware Avi Load Balancer

Issue/Introduction

Configuration local and remote backups stop working and files are not being created.

The backup has a timestamp column on the page as well as in the file name.  If the backup file timestamp is not accurate per the configuration frequency and the actual date then this is a symptom of this issue.

You can further confirm with the error in the logs below:

('Failed: https://localhost/login Status Code 429 msg {"error":"Lock acquisition timeout"}', <Response [429]>)

Exception in /var/lib/avi/log/scheduler.log

You will also see error message "lock acquisition timeout" in /var/lib/avi/log/apiserver* file(s).

Other Symptom(s):

The GUI/CLI/API will be unavailable and results in HTTP 500 internal server error.

GUI: Login Page

CLI:

admin@controller:~$ shell
Login: admin
Password:

The controller at ['https://localhost'] is not active. If this is not the controller address, please restart the shell using the --address flag to specify the address of the controller.
Unknown error
The controller at ['https://localhost'] is not active. If this is not the controller address, please restart the shell using the --address flag to specify the address of the controller.
Unknown error
The controller at ['https://localhost'] is not active. If this is not the controller address, please restart the shell using the --address flag to specify the address of the controller.
Unknown error

 

If using LDAP for controller management access (GUI), you will receive the following error from remote user(s):

Login error in the GUI:

The error messages "Lock acquisition timeout" and "ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)" will be observed in the /var/lib/avi/portal-auth.log for the user's failed login.

2025-11-21T09:06:47.616Z        I  3664025      authserver/authenticator.go:171  [T-ID=26c55996] [AuthServer Authenticate]: Authentication started
2025-11-21T09:06:47.618Z        I  3664025      authserver/authenticator.go:229  [T-ID=26c55996] [AuthServer authenticateBackends]: Available auth backends - [2 3 5 6]
2025-11-21T09:06:47.618Z        I  3664025      authserver/authenticator.go:233  [T-ID=26c55996] [AuthServer authenticateBackends]: Trying KEYSTONE authbackend
2025-11-21T09:06:47.618Z        I  3664025      authserver/authenticator.go:233  [T-ID=26c55996] [AuthServer authenticateBackends]: Trying LDAP authbackend
2025-11-21T09:06:47.622Z        I  3664025      authserver/authenticator.go:233  [T-ID=26c55996] [AuthServer authenticateBackends]: Trying LOCAL authbackend
2025-11-21T09:06:47.905Z        I  3694445      authserver/authenticator.go:233  [T-ID=26c55996] [AuthServer authenticateBackends]: Trying TOKEN_BACKEND authbackend
2025-11-21T09:06:51.895Z        E  3694445      infra/transaction.go:56 Lock acquisition timeout
2025-11-21T09:06:51.895Z        E  3694445      utils/auth_rules.go:819  [T-ID=1465582c] [AUTH RULES]: UserActivity Protobuf2Model save failed with Lock acquisition timeout
2025-11-21T09:06:51.896Z        E  3694445      ldap_auth/ldap.go:270    [T-ID=1465582c] [LDAP AUTH]: user example, Not authorized.
2025-11-21T09:06:51.901Z        E  3694445      infra/transaction.go:56 ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
2025-11-21T09:06:51.901Z        E  3694445      apihandlers/custom_auth_apihandlers.go:114       [T-ID=1465582c] [handleLoginException]: UserActivity DB save failed with ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)!
2025-11-21T09:06:51.908Z        E  2149013      apihandlers/base_apihandlers.go:521      [T-ID=1465582c] Error encountered during /login - ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)

Environment

Affected Version(s)

30.1.x

30.2.1 - 30.2.1-2p6

30.2.2 - 30.2.2-2p6

30.2.3

Cause

This issue was conformed to be a corner case issue (race condition) where session cleanup jobs and the API based login flows create a deadlock resulting in subsequent API failures with error "Lock acquisition timeout" in heavy scale setups.

Resolution

Please upgrade to the fix version or latest Avi GA release for permanent resolution.

ID: AV-234913

Fix Version(s):

30.2.3-2p1

30.2.4 and latest 30.2.x releases

31.1.1 and latest 31.1.x releases

Workaround:

Run below command to restart services in each controller node sequentially: 

systemctl stop apiserver.service

systemctl stop authserver.service

systemctl stop aviportaljobmanager.service

systemctl stop aviportal.service