Impact of shuting down PAM during Stage process

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

This article explains the risks associated with rebooting or shutting down Privileged Access Manager (PAM) nodes once the upgrade "staging" process has commenced. It addresses why cluster convergence may fail if the environment is powered down during this phase.

Environment

PAM 4.3.0 and up

Cause

The "Staging" process in PAM initiates structural changes and preparations on the nodes in anticipation of the full upgrade. If a node is rebooted during this transition, the local configuration may not align with the cluster's expected state, preventing successful convergence. PAM typically generates a UI warning stating that reboots should not occur while an upgrade process is in progress.

Symptoms

• The upgrade staging process for a version (e.g., 4.3.1) is started.
• Nodes are subsequently shut down or rebooted before the "Prepare" or "Upgrade" phases are triggered.
• Upon restart, the cluster fails to converge or nodes remain in an inconsistent state.

Resolution

• Do not shut down or reboot PAM nodes once the upgrade staging process has begun.
• Ensure the upgrade process (Prepare and Upgrade) completes fully before performing any power actions.
• If a reboot occurs accidentally during staging and the cluster fails to converge:
  1. Roll back all nodes to the VM-level snapshots taken immediately prior to starting the upgrade process.
  2. Restart the upgrade process end-to-end.
  3. Verify cluster health before initiating the staging phase again.