Post upgrade to CA PAM 4.2.0 appliances having SiteMinder integration configured do not start

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

After upgrading an appliance from any of the supported versions to version 4.2.X, it is no longer possible to access the appliance, even if it is pingable

Trying to access it results in error code 503 Service not available. 

Environment

CA PAM all supported versions upgraded to version 4.2.X

Cause

This issue will occur if in the version prior to upgrade, the CA SiteMinder integration had been performed, even if it is currently disabled.

Whenever the integration is performed, CA PAM modifies its apache web server configuration file to include loading a set of libraries specific of the SiteMinder integration. These lines remain even if CA SSO integration is disabled in the CA PAM appliance console 

In version 4.2.0 the CA SiteMinder integration has been deprecated and the libraries used for it no longer exist in the system (consequently also the SiteMinder entry has disappeared from under the Symantec Modules under the Configuration page of the product).  However, the upgrade process does not take this into consideration and it does not remove the SiteMinder-related entries from the apache configuration file

As a result, with CA PAM starts, its apache web server tries to load a set of libraries which no longer exist in the system and can therefore not start. This makes the web server not available and the error code referenced above is obtained when trying to contact CA PAM.

Resolution

Once the appliance has been upgraded to version 4.2.X there is no solution other than having Broadcom support access the console and perform some manual operations to correct the apache configuration file.

This in turn requires the Remote Symantec PAM Debugging Services to be enabled under the Configuration --> Diagnostics --> System page and a valid PAM_SUPPORT_SSH_DEBUG patch installed under Configuration --> Diagnostics --> Upgrade. Please remember that the PAM_SUPPORT_SSH_DEBUG patches change every month so make sure to have a recent version. If you don't please contact Broadcom Support and request for it to be uploaded. Take into account that upon installation of a new ssh debug patch Remote Debugging Services must be switched off and on to have it load the new keys into memory.

Enabling Remote Diagnostics and having an PAM_SUPPORT_SSH_DEBUG patch installed is one of the best practices that Broadcom Support recommend as per the following document:

How to Prepare for a Privileged Access Manager Upgrade

If you do not have the Remote Symantec PAM Debugging Services option enabled there is unfortunately no way to access the appliance to remediate the problem once the upgrade is completed.

If you are in this situation, the only viable option is to revert back a snapshot of the machine to your previous CA PAM version, install a PAM_SUPPORT_SSH_DEBUG patch and enable Remote Symantec PAM Debugging Services, and  open a ticket with Broadcom Support so that they access your appliance(s) and perform the necessary operations prior to upgrading. There are some 4.1.X versions for which a patch exists that will avoid a manual correction, but you need to check with Broadcom Support if your appliance can be corrected this way. Otherwise a remote session with support will be required and manual changes implemented in every appliance where SiteMinder integration may have been configured in the system being upgraded.