Randomly Service Outage on gateway

Products

• CA API Gateway
• API SECURITY
• CA API Gateway Precision API Monitoring Module for API Gateway (Layer 7)
• CA API Gateway Enterprise Service Manager (Layer 7)
• STARTER PACK-7
• CA Microgateway

Issue/Introduction

We applied the last platform update and latest CR4  patch to production. Then, we started to observe some errors in the production environment.When we disable and enable the corresponding service, the problem is resolved but after a while it occurs again. a similar problem is also seen in the test environment but root cause cannot be found, . There is a problem that seems related to reaching out to the service cache.

 

 

Environment

Release : 9.4

Component : API GATEWAY

Cause

This is related to the following udp data sockets error in the ssg log
" assertion for com.l7tech.external.assertions.snmptrap.SnmpTrapAssertion: java.net.SocketException: maximum number of DatagramSockets reached"
This could occur when the snmp trap assertion is triggered to many times in parallel

If this "send snmp trap" assertion I used in a global policy like message received it impacts all services randomly without clear indications why the service failed

Resolution

The solution would be to increase the datagram sockets java can allocate at the same time.

Increase the limit of Datagram sockets by adding the following to the /opt/SecureSpan/Gateway/node/default/etc/conf/system.properties file and restarting the Gateway process:

sun.net.maxDatagramSockets=1000