PAM VMs not accessible due to machine id conflict from cloning post initial boot

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

We added two PAM VMs to a cluster, but they are not working properly. It turned out that they both had the same machine id even though the network interfaces (MAC address/IP) are different. A reboot doesn't change that. Can this problem be fixed, or do we need to replace one of the two nodes?

Cause

In PAM 3.X releases prior to 3.3x and 3.4.x PAM virtual machines recalculated the machine ID, which is needed e.g. when requesting a license, at boot time. Adding an interface, or changing the MAC address of an existing interface configured for the VM, would cause a change in the machine ID. This did not cause a problem with data integrity in a cluster environment, because local data in the database did not get replicated across the cluster. However, with the change in database replication technology from PAM 3.3 on, the machine ID is used extensively in the PAM database to distinguish database entries that are specific to individual cluster nodes such as session logs, certificate settings, session recording settings etc. For that reason the machine ID no longer changes after initial PAM startup. This is not a problem if VMs are deployed following our documented procedure, e.g. on page Deploy the VMware OVA Template. We explicitly state "Verify that the Power on after deployment check box is not selected", because we need the subsequent cloning of the VM to occur before the VM is powered on for the first time. If you clone a VM after the initial boot time, i.e. after the machine ID has been created, you will end up with VMs using the same machine ID, which causes problems with licensing and prevents the nodes from working in a cluster environment.

Resolution

Make sure to create new PAM VMs either directly from the PAM OVA file, or from a template based on the OVA file that was never powered on, as documented in the link mentioned above. When you add a VM to an existing cluster, check the machine ID on all current cluster nodes and verify that the ID of the new VM is unique. If you have the problem already, and there are good reasons why you cannot just replace one of the VMs with a new instance, PAM support may be able to fix the problem with SSH access to the appliance. But this option should be attempted only as a last resort, if no other options are available.