Service Engine creation fails with reason : service engine creation timed out

Products

• VMware Avi Load Balancer

Issue/Introduction

• New Service Engine (SE) VM creation fails or times out.
• Virtual Services remain in a OPER_CREATING state with reason: "Waiting for Service Engine <SE> creation"
• Below events are noticed on Avi LB under Operations > Events > All Events 
  • CREATE_SE_FAIL with reason : service engine creation timed out
  • CC_SE_CREATION_FAILURE with error string: Post \"https://<vCenter_address>/rest/com/vmware/vcenter/ovf/library-item/id:<id>?~action=filter\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Environment

Avi Load Balancer

NSX or vCenter cloud

Cause

This issue occurs when the Avi Cloud Connector sends an API request to the vCenter and the vCenter API fails to respond within the allocated timeout period. 

The error noticed in Avi under /var/lib/avi/log/cc_agent_go_<cloud>.log

    [<SE-Name>] Post "https://<vcenter_address>/rest/com/vmware/vcenter/ovf/library-item/id:<id>?~action=filter": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

 

Resolution

• The error mentioned in the above log line signifies some limitation/issue hit at vCenter. Engage vCenter Support team to check the health of vCenter.
   
  1. Check vCenter Server health and load (CPU, memory, and disk usage on the vCenter Server Appliance), as high resource utilization can cause the API to respond slowly or time out.
  2. Restart the vCenter Server services, or reboot the vCenter Server Appliance if the issue persists. This is typically performed during a maintenance window, as it can briefly disrupt other vCenter-dependent operations.
  3. After vCenter comes back online, retry Service Engine creation from the Avi Controller and confirm that the CREATE_SE_FAIL and CC_SE_CREATION_FAILURE events no longer occur.