Random password update failures with error code 1604 during scheduled job for AD accounts

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

More than 15% of privileged accounts password rotations are failing in each password rotation job with error code 1604( Authentication failed) even though we are using an AD service account for changing the credentials for the Privileged Accounts, and the service account always is in a Verified state. A different set of accounts is affected in each job, we cannot identify any pattern.

Environment

Affects all PAM releases as of November 2023.

Cause

A load balancer address was configured in PAM as device for the Active Directory target application. This load balancer could direct calls to domain controllers in two different datacenters. Whenever PAM updated the password on a domain controller in datacenter 1, and then was redirected to datacenter 2 for password verification, the latter would fail because AD had not replicated the new password to the other datacenter yet. When PAM fails to verify the new password, it will regard the update as failed and retain the old password.

Resolution

Either configure PAM to connect to AD controllers in one datacenter only, or even to a specific domain controller, or add source IP persistence to the external load balancer so that it will direct all connections from a given PAM server to the same AD controller within the time interval specified in the source IP persistence configuration.