PAM-CMN-3234 errors on new cluster nodes after removing HSM configuration and upgrading the master

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

We are running a 3.2.6 cluster with HSM as crypto provider. We are trying to upgrade to 3.4.1 and switch to OpenSSL. There is no noticeable problem on the 3.2.6 node after removing the HSM configuration. We already had other PAM instances ready running 3.4.1 and configured in a cluster. We made the upgraded node master (first node in the primary site) and started the cluster. But after the cluster was up, the dashboard showed zero target accounts on the new nodes, we could not view credentials, and the session log was full of "PAM-CMN-3234: The HSM is not functioning properly with PKCS11 result: 224, doDecrypt() failed, will do library reload after 10 seconds then retry!" errors. The 3.4.1 nodes never had HSM configured.

Environment

Release : 3.4

Component : PRIVILEGED ACCESS MANAGEMENT

Cause

After removing the HSM configuration on a PAM server, you are asked for a reboot. The conversion of passwords in the PAM database from the old encryption method to the new one only starts AFTER the reboot and it can go on for a very long time, particularly if there are many target accounts configured in PAM, and many of those accounts have a long password history stored in PAM. The PAM UI does not provide information on the progress of password re-encryption, it is meant to be transparent to the PAM admin. The node where the encryption method is changed will have no problem retrieving passwords that have not been converted yet. But other nodes loading the same database will only know how to decrypt the already converted passwords.

Resolution

After removal of the HSM configuration it is required to wait for the conversion to complete before taking administrative actions, such as an upgrade or addition of new cluster nodes. If there is a requirement to do both during a single maintenance period, you would want to engage PAM Support. Support can monitor the conversion progress with SSH access to the appliance. There are also options to speed up the conversion by removing all but the latest password history entries for each target account, and reduce sleep times that by default pause conversion for 15 minutes at a time in order to not cause performance problems in the case where the only activity is the change in the encryption method, and user traffic is meant to be restored right after.