Configuration updates blocked on Cluster after a Leader Change on Avi

Products

• VMware Avi Load Balancer

Issue/Introduction

This issue may manifest in many ways, most commonly affecting any configuration updates on the Cluster.

Symptoms:

1. Create/Update of objects are blocked and show below error
   
   
   
   
2. Remote User Authentication may fail
   
   /login API call made by remote user fails with 500 error "cannot execute SELECT FOR UPDATE in a read-only transaction (SQLSTATE 25006)"
   
   

   Octavius.ERROR
   
   2025-03-05T15:48:12.583Z E 582420 octavius/octavius_db.go:53 [T-ID=fd140c83] :useractivity-e5547146-c46e-4342-9d1e-7c1f2eb49f4e(#####@#####.com) -> DB save failed with ERROR: cannot execute SELECT FOR UPDATE in a read-only transaction (SQLSTATE 25006)!
   2025-03-05T15:48:12.584Z E 571671 octavius/octavius_grpc_handlers.go:451 [T-ID=fd140c83] :useractivity-e5547146-c46e-4342-9d1e-7c1f2eb49f4e(#####@#####.com) (#0 of Macro op(API_POST)) failed with ERROR: cannot execute SELECT FOR UPDATE in a read-only transaction (SQLSTATE 25006)
   2025-03-05T15:48:12.584Z E 571671 octavius/octavius_grpc_handlers.go:643 [T-ID=fd140c83] Errors encountered in MacroAPIAction, rpcresponse: [action_status:false response_status:<rpc_status:500 error_message:"ERROR: cannot execute SELECT FOR UPDATE in a read-only transaction (SQLSTATE 25006)" > response_data:<action_type:API_POST data:<useractivity:<uuid:"useractivity-e5547146-c46e-4342-9d1e-7c1f2eb49f4e" name:"[email protected]" failed_login_attempts:0 last_login_timestamp:"2025-03-05 15:48:12" concurrent_sessions:0 > > metadata:<token:"3b31182f-1026-4df1-94ca-4950c9158cc8" skip_semantic_checks:true skip_optional_semantic_checks:false run_alternate_checks:false req_meta_data:<user:"" path:"/login" scheme:"https" host:"#####@#####.com" user_agent:"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" client_ip:"#.#.#.#" > > > api_perf_stats:<name:"DbSave" start:1741189692583408527 end:1741189692583857897 > api_perf_stats:<name:"ProcessObjectNoTxn" start:1741189692583315263 end:1741189692583919959 > ]

Environment

Version 22.1.x, 30.2.x

Cause

After a leader change, in some scenarios the octavius service is not stopped after the previous leader becomes the follower. This is causing the service to be incorrectly pointing to the previous leader, blocking any updates to the db.

Resolution

Workaround:

Go to Administration > Controller > Nodes to identify the leader node

Reboot the Leader Node VM 

or 

Run the command "warm reboot" on the CLI

Bug ID: AV-218791

Release Notes: Configuring an Avi deployment via API / CLI / UI may fail after a Controller leader change.

Fix Versions: 22.1.7-2p3, 30.2.3