How Dynatrace impacts the API Gateway and how to disable Dynatrace

Products

• CA API Gateway
• API SECURITY
• CA API Gateway Precision API Monitoring Module for API Gateway (Layer 7)
• CA API Gateway Enterprise Service Manager (Layer 7)
• STARTER PACK-7
• CA Microgateway

Issue/Introduction

• The gateway node has performance issue and need to restart every few days
• There is out-of-memory error on metaspace

java.lang.OutOfMemoryError: Compressed class space

• The gc log shows Over 80% of GC is caused by metaspace ( "Metadata GC Threshold" + "Last ditch collection")
• The vmstat shows user mode CPU spike every 10 seconds

Environment

Release : 9.4

Component : API GATEWAY

Cause

The Dynatrace was installed on gateway nodes.

The problem is gone after disable the Dynatrace one agent.

Resolution

Disable Dynatrace entirely -- backup and remove file /etc/ld.so.preload (The dynatrace one agent is loaded for every process via the ld.so.preload)

# cat /etc/ld.so.preload
/$LIB/liboneagentproc.so

 

Alternately, if still want to use Dynatrace for monitoring, would need the Dynatrace expert to tune the Dynatrace until it doesn't impact the gateway.