Configuration back up scheduler failing with error "Lock acquisition timeout" or GUI/CLI inaccessible HTTP 500 response

Products

• VMware Avi Load Balancer

Issue/Introduction

Configuration local and remote backups stop working and files are not being created.

The backup has a timestamp column on the page as well as in the file name.  If the backup file timestamp is not accurate per the configuration frequency and the actual date then this is a symptom of this issue.

You can further confirm with the error in the logs below:

('Failed: https://localhost/login Status Code 429 msg {"error":"Lock acquisition timeout"}', <Response [429]>)

Exception in /var/lib/avi/log/scheduler.log

You will also see error message "lock acquisition timeout" in /var/lib/avi/log/apiserver* file(s).

Other Symptom:

The GUI/CLI/API will be unavailable and results in HTTP 500 internal server error.

GUI: Login Page

CLI:

admin@controller:~$ shell
Login: admin
Password:

The controller at ['https://localhost'] is not active. If this is not the controller address, please restart the shell using the --address flag to specify the address of the controller.
Unknown error
The controller at ['https://localhost'] is not active. If this is not the controller address, please restart the shell using the --address flag to specify the address of the controller.
Unknown error
The controller at ['https://localhost'] is not active. If this is not the controller address, please restart the shell using the --address flag to specify the address of the controller.
Unknown error

Environment

Affected Version 30.2.1

Cause

This issue was conformed to be a corner case issue (race condition) where session cleanup jobs and the API based login flows create a deadlock resulting in subsequent API failures with error "Lock acquisition timeout"

Resolution

Enhancements to prevent this issue will be added to the future GA releases.

Workaround:

Restart apiserver and aviportal on the controller leader node.

systemctl restart apiserver.service

systemctl restart aviportal.service