High replication delay in Identity Manager pgpool cluster when SSH public key authentication is disabled

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• VMware Identity Manager Health is Critical in Aria Suite Lifecycle Manager.
• REMEDIATE operation fails with errors similar to:

Error Code : LCMVIDM74055

Unable to perform pcp recovery on the host. Check if there exists a primary node in the set-up. Refer to VRSLCM log for additional details and retry.

Unable to recover all the postgres nodes which are marked down. Ensure the nodes are powered on and delegateIp is assigned to primary node.

• Listing pool nodes using command su root -c "cat /usr/local/etc/pgpool.pwd|/opt/vmware/vpostgres/current/bin/psql -h delegateIP -p 9999 -U pgpool postgres -c \"show pool_nodes\"" displays high replication delay:

node_id | hostname  | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change
---------+---------------+------+--------+-----------+---------+------------+-------------------+-------------------+-----------
0     | XX.XXX.XX.XX | 5432 | up   | 0.333333 | primary | 1        | true            | 0               | 2023-12-01 10:23:42
1     | XX.XXX.XX.XX | 5432 | down | 0.333333 | standby | 0        | false           | 495885351589    | 2023-12-01 10:39:15
2     | XX.XXX.XX.XX | 5432 | down | 0.333333 | standby | 1        | false           | 1178392         | 2023-12-01 10:39:18

• Manual steps from KB 322709 did not resolve the issue.

Environment

VMware Identity Manager 3.3.x

Cause

This issue can be caused if SSH public key authentication is disabled between vIDM nodes in the cluster.

Resolution

To resolve this issue, verify and ensure the SSH public key authentication is enabled on all vIDM nodes.

1. Open an SSH session to vIDM node 1 and authenticate using root credentials.
2. From this node, open SSH session to node 2:

root@vidm1 [ ~ ]# ssh root@vidm2
Welcome to Photon 3.0 (\m) - Kernel \r (\l)
root@vidm2's password:

Note: if you are prompted to enter the password like in this example, it means the public key authentication is disabled.

3. Open /etc/ssh/sshd_config file in a text editor:

vi /etc/ssh/sshd_config

4. Locate the line containing PubkeyAuthentication no.
5. Update it to PubkeyAuthentication yes and save the file.
6. Restart SSH daemon:

systemctl restart sshd

7. Retry the SSH login and verify you can login successfully without entering the password:

root@vidm1 [ ~ ]# ssh root@vidm2
Welcome to Photon 3.0 (\m) - Kernel \r (\l)
Last login: Thu Dec  7 09:52:21 2023 from <IP address>
#################################################################################
# Attention:                                                                    #
#                                                                               #
# Do not perform a manual reboot of the appliance as it can affect the database #
# clustering. Instead use Power OFF/ON from vRealize Suite Lifecycle Manager    #
# by navigating to the global environment page.                                 #
#                                                                               #
#################################################################################
root@vidm2 [ ~ ]#

8. Repeat steps 3-7 on remaining nodes in the cluster.
9. Navigate to Aria Suite Lifecycle Manager and retry the REMEDIATE option.