Rest API call GET targetApplications fails for devices with many target applications

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

We have an team that has devices with many target applications on them, which they sometimes need to enumate through the "GET /api.php/v1/devices.json/{id}/targetApplications" call.  One device has 725 applications, and used to return these without issue.  Now, devices with ~175 applications time out this call.  While the team is not certain of the exact timing, it appears that the troubles began with the upgrade from 4.0.2 to 4.1.1, or possibly shortly thereafter.

We know that devices with 9 applications still complete the call, but using limit=10 does not make the call for devices with 175 applications return in time.

We are not using Secrets Management yet, but we also noticed that we cannot create secret vaults with secrets while this problem persists.

Environment

Privileged Access Manager 4.1.1+

Cause

The problem was not caused by the upgrade to 4.1.1, but by a subsequent change in the Docker CIDR, see documentation page Configure Docker Network Settings. PAM runs a local firewall to restrict access to the database. The network startup scripts add a rule to allow connections from Docker containers, but this rule was not updated when the new CIDR was configured. One of the docker containers running on the PAM appliance provides a custom connector to support Secrets Management, and is called into while retrieving application details. These calls would time out because of the missing firewall rule, causing delays of several seconds per target application retrieved. The sum of these delays could exceed the timeout limit of the Rest API call for long target application lists.

Resolution

Restarting the network using the Restart Networking button at the bottom of the Configuration > Network > Network Settings page is sufficient to resolve the problem, as is an appliance reboot. This should be done one node at a time in an active cluster. The network restart is less disruptive than a node reboot, but still is likely to cause problems for active user sessions.

The only way to resolve the problem without impacting PAM users would be to allow PAM Support to add a firewall rule for the new Docker CIDR using SSH debug access to the PAM appliances. If you don't have times of no or low user activity and are affected by this problem, open a case with PAM Support.