PAM Utility -- After a Reboot, Most Utility Appliance Services Cannot Start -- (EE:EE:EE:EE:EE:EE)

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

After the Utility Appliance was rebooted, only the pam-a2a and pam-dh services are started.

 

The stopped services have the same error saying "Failed to fetch activemq password."

 

Tomcat logs do show  <macaddr>EE:EE:EE:EE:EE:EE</macaddr>

Environment

Privileged Access Manager 4.0.x, 4.1.x with one or more Utility Appliances

Cause

The ActiveMQ password is stored in the PAM appliance, the other services on the Utility Appliance use the pam-a2a service to get the password. The pam-a2a service was running but had the following errors in its log.

2022-07-07T13:11:23.783199410Z WARNING: Thu July 07 13:11:23.782 UTC 2022 ClientService::loginToCSPMServer. start
2022-07-07T13:11:24.085806754Z WARNING: Thu July 07 13:11:24.085 UTC 2022 KeyService::clientFixedKeyLogin. No key value detected
2022-07-07T13:11:24.085928362Z WARNING: Thu July 07 13:11:24.085 UTC 2022 ClientService::loginToCSPMServer. Failed to perform CSPM Server login. Retrying...
2022-07-07T13:11:24.086020344Z ClientService::loginToCSPMServer. Failed to perform CSPM Server login. Retrying...
2022-07-07T13:11:27.133406859Z WARNING: Thu July 07 13:11:27.133 UTC 2022 KeyService::clientFixedKeyLogin. No key value detected
2022-07-07T13:11:30.176110842Z WARNING: Thu July 07 13:11:30.175 UTC 2022 KeyService::clientFixedKeyLogin. No key value detected
2022-07-07T13:11:33.220354947Z WARNING: Thu July 07 13:11:33.220 UTC 2022 KeyService::clientFixedKeyLogin. No key value detected

When an A2A agent communicates with the PAM appliance, it uses the server's MAC address to generate a fingerprint to confirm its identity. If there is any change to the MAC address, it causes the fingerprint to change and the A2A agent will no longer be able to communicate.

In this case, the number of virtual interfaces on the Utility Appliance changed. Since each virtual interface has its own MAC address, the fingerprint used by the pam-a2a service changed and it could not communicate with the PAM appliance.

Resolution

Development is looking into the code to change this behavior in a future release.

As a workaround, remove the Utility Appliance from the Utility Group and add it back in. This will redeploy the services on the Utility Appliance and re-register the A2A client in PAM.