SSH connection fails from PAM to targets after patching OS

Products

• CA Privileged Access Manager - Cloakware Password Authority (PA)
• PAM SAFENET LUNA HSM
• CA Privileged Access Manager (PAM)

Issue/Introduction

After applying the latest quarterly Oracle patch set to a cluster environment of six target servers,  users can no longer connect to three of six servers using SSH auto-login. 

The quarterly patch had updated the version of openssh, so this was suspect. 


 

Cause

Root cause:  The password history for the accounts used for auto-connect showed that the EXPPWD process had updated the password during the maintenance window of the OS being patched.

The change in openssh versions had nothing to do with the problem.

The TargetAccountExpiredPassword process will not change the password when there is a failure to connect to the target.

The method followed to shut down the targets and patch them and return them left a window of opportunity where the target systems were connected during the time that the passwords required updating. 

Environment

PAM 3.2.3
Target servers are Oracle Solaris 11g database servers 

Resolution

Retrieved the new password from PAM and updated the problem accounts on the target system.
 

Additional Information

Best Practices:

When target systems will be removed for patching, suspend account updates to those targets.
1.  Check that no scheduled jobs will be running during the patching window.
2.  Under Settings - Credential Manager - General Settings, uncheck the box for 'Automatically Update Expired Passwords'

The TargetAccountExpiredPassword process runs every 12 hours and will catch the accounts and update them when the setting is enabled again.