Very high HTTP/HTTPs open connections Alarm

Products

• VMware NSX

Issue/Introduction

Title: Alarm for very high HTTP/HTTPs open connections.

Event ID: load_balancer. l7_open_connections_overload


Alarm Description:

• Purpose: To inform users that too many HTTP/HTTPs open connections.
• Impact: Some clients will experience connection rejections when the limit is reached.

Environment

VMware NSX

Cause

This indicates that the application services are scaling up.

Resolution

Steps to resolve
For 9.0.0 and higher

Recommendation Action:

1. To manage increased traffic, it is recommended to scale up the load balancer to support higher concurrent HTTP/HTTPS connections. However, resizing on the current Edge may fail due to capacity limitations, necessitating relocation to another edge node followed by resizing.
2. To determine the optimal course of action, provide the load balancer ID to invoke the Load Balancer Scale Runbook API at /infra/sha/runbook-invocations/<invoke_id> with a unique <invoke_id>, e.g. lb_scale_4b17dca6. This will deliver detailed action plans for resizing the load balancer, including any required relocation and resizing steps.
3. You can manually relocate the load balancers by invoking PATCH /policy/api/v1/infra/tier-1s/<tier-1-id>/locale-services/<locale-services-id> with the identified Edge node ID in the request body.
4. If you fail to identify a suitable Edge node, deploy a new Edge node and migrate the load balancer services from the current Edge nodes to the newly deployed node.

Maintenance window required for remediation? Yes