Appliance in Unresponsive State with browser showing ERR_EMPTY_RESPONSE

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

An appliance that is a member of a secondary site that is unresponsive via web browser or client. The HTTPS port appears to be open, but the browser shows "This page isn't working", <address> didn't send any data, ERR_EMPTY_RESPONSE

This appliance typically has a lot of user activity with recorded access sessions. The problem was observed several weeks after upgrading to 4.2.0.

Environment

PAM 4.2.0-4.2.2

Cause

The node gradually ran out of memory. The memory was consumed by processes that had been started when users with access policies that included session recording logged on to PAM. On user logout one process got hung trying to exit while still having a significant amount of memory allocated. This happened for login sessions from users that had access policies configured with session recording enabled. It didn't matter whether or not the user actually launched an access session. The node where the problem was observed had 64GB of RAM and 16k of these processes running, which used up all memory not in use by other PAM services. This prevented PAM from responding to new incoming connections.

Resolution

PAM 4.2.0: Hotfix 4.2.0.64

PAM 4.2.1: Hotfix 4.2.1.10

PAM 4.2.2: Hotfix 4.2.2.01

The problem will be fixed in 4.2.3+.

Additional Information

This problem takes time with RAM usage gradually moving towards 100%. Monitoring RAM usage on the dashboard, via Rest API call (GET /cspm/ext/rest/dashboard/clusterSiteInfo) or SNMP polling should allow you to catch the problem before it makes the node inaccessible.