CSR Certificates Managed via Certificate Management Profile Enter Continuous Renewal Loop in VMware AVI Load Balancer.

Products

• VMware Avi Load Balancer

Issue/Introduction

• On the VMware Avi Load Balancer Controller, certificates managed through the Certificate Management Profile may become stuck in a rapid renewal loop.
• This occurs when a Certificate Signing Request (CSR) fails to complete successfully, leading the system to trigger repeated renewal attempts every few seconds.
• These failures are highly visible in the system alerts and can lead to alert fatigue and unnecessary system overhead.
• 

Environment

VMware AVI load balancer

• 22.1.5

Cause

• During the certificate renewal process, the Controller processes all certificates returned in the CA response, including the full trust chain. Each certificate and its corresponding chain elements are created and updated as separate, individual objects.
• These objects are transmitted to the backend (Octavius) for processing.
• However, due to the asynchronous nature of these requests, they may occasionally be processed out of order. If a leaf certificate is processed before its parent/root certificate, a dependency mismatch occurs, resulting in a "Record Not Found" failure.
• On the controller leader node under the var/log/upstart/aviportaljobmanager.log you will notice the error below:

0218 13:41:44.038281    E  7016         jobmanager/jobmanager.go:659    Error when renewing cert. Error message: Encountered an error on POST request to URL https://localhost//api/sslkeyandcertificate/sslkeyandcertificate-######-#####-#####-#####-#######/renew: HTTP code: 400; error from Controller: map[error:record not found traceback:Traceback (most recent call last):                                                File "/opt/avi/python/bin/portal/api/views_ssl_custom.py", line 196, in post_transaction                                                                                                                             send_octavius_request(ssl_nodes_octavius)                                                                                                                                                                        File "/opt/avi/python/lib/avi/rest/octavius.py", line 297, in send_octavius_request                                                                                                                                  resp = send_octavius_grpc(macro_req_pb, 'MacroAPIAction', ignore_error=ignore_error)                                                                                                                             File "/opt/avi/python/lib/avi/rest/octavius.py", line 158, in send_octavius_grpc                                                                                                                                     check_octavius_response(r)                                                                                                                                                                                       File "/opt/avi/python/lib/avi/rest/octavius.py", line 173, in check_octavius_response                                                                                                                                raise OctaviusException(status_code, error_msg)                                                                                                                                                                avi.rest.error_list.OctaviusException: record not found                                                                                                                                                            ]   

Resolution

Resolution

• The issue has been resolved in the following Vmware AVI Load Balancer versions.
  • 22.1.6
  • 30.2.1

Workaround

• To stop the renewal loop:
  • Identify the specific certificate that is failing renewal.
  • Delete the certificate from the Controller.
  • Once deleted, manually re-issue the certificate or ensure the full trusted chain is manually uploaded to the account before attempting to re-enable automated management to prevent future reference errors.