Layer7 API Gateway Thread Leak

Products

• CA API Gateway

Issue/Introduction

There was an outage on one of our Gateway servers on production caused by Gateway process hitting the maximum number of subtasks allowed (sysctl DefaultTasksMax, 4915).

This behaviour was unexpected because the traffic and policy processing pattern has always been carried with around 2.4k threads

Environment

11.1

Cause

Gateway process reaching the DefaultTasksMax limit imposed by systemd on the Debian operating system used in Gateway 11.x.is the main cause for the issue 

By default, this limit is often set to 15% of the kernel's PID maximum, which equates to exactly 4915 threads. When your traffic spikes or requires threads beyond this limit, the OS prevents the Gateway from creating new threads, leading to the instability you experienced.

 

Resolution

To resolve this and prevent future accumulation issues, please follow these steps:

1. Increase the systemd Task Limit:
   • Edit the file /etc/systemd/system.conf.
   • Locate or add the entry DefaultTasksMax.
   • Set this to a higher percentage, such as DefaultTasksMax=30% or 60%, depending on your required concurrency.
   • A system reboot is required for this change to take effect.
2. Review Cluster Properties: Check your io.httpCoreConcurrency and io.httpMaxConcurrency cluster-wide properties. If these are set close to the 4915 limit, the Gateway will hit the OS cap immediately upon reaching those values.
3. Monitor oioEventLoopGroup Threads: These threads are related to internal asynchronous I/O handling.