Alerts in VMware Aria Suite Lifecycle about vIDM cluster health with Secondary VMware Identity Manager nodes going 'down' occasionally

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• This article applies to VMware Aria Suite Lifecycle 8.14 or later with Cluster-Auto-Recovery enabled for Workspace ONE Access (VMware Identity Manager vIDM 3.3.x).
  
  
  • VMware Aria Suite Lifecycle 8.14 Release Notes
• Secondary nodes have replication delays over 1000 bytes with the primary node and are intermittently down.
• Alerts are visible in VMware Aria Suite Lifecycle about vIDM Postgres Cluster health.
• The VMware Identity Manager service is experiencing some issues. End user can still log in and launch apps. banner on vIDM UI.
• vIDM Dashboard shows GREEN status for all components and services operational otherwise.
• Triggering inventory sync in Aria Suite Lifecycle for the vIDM environment may return the error:

  Error Code: LCMVIDM73107
  Postgres status of the VIDM cluster set-up is not good.
  Refer to logs for additional details and retry.
  Postgres health status of the cluster set-up is not good-
  Node(s) ###.###.###.###.###.###.###.###, have a replication
  delay with respect to the master node... Please refer to KB:
  https://kb.vmware.com/s/article/7508

Environment

VMware Identity Manager 3.3.x

Cause

Replication delays can occur due to underlying network latency. The VMware Identity Manager cluster auto-recovery service is configured with a default replication delay threshold of 1000 bytes. Beyond this threshold, auto-recovery triggers, taking the secondary nodes offline ("down") to synchronize their data with the primary.

VMware Aria Suite Lifecycle performs an hourly health check on VMware Identity Manager appliances. If replication delays or node "down" events occur during these health checks, VMware Aria Suite Lifecycle issues a health status notification.

Resolution

To resolve the issue, correct the underlying network issue that is causing the replication delays between Identity Manager nodes.

Workaround:

To work around the issue, increase the auto_recovery_replication_delay_threshold setting.

1. Find the typical replication delays on this setup:

   cd /var/log/pgService
   zgrep "SECONDARY_1_REPLICATION_DELAY" * | cut -d '=' -f2 | uniq | sort -rn | head -n50
   # Similarly for SECONDARY_2
   zgrep "SECONDARY_2_REPLICATION_DELAY" * | cut -d '=' -f2 | uniq | sort -rn | head -n50

2. Based on the highest delays reported above, configure the new threshold in /usr/local/etc/lcm-pgpool.conf.
   For example:

   auto_recovery_replication_delay_threshold=100000

   Note: 100000 is just a reference value. It is recommended to set the actual threshold value based on the maximum values seen in the output of the commands listed in step 1.

3. Perform this change on all the Identity Manager nodes in the cluster.

Additional Information

Impact / Risks

Replication delays occur due to a slow underlying network or other unforeseen network characteristics. To alleviate notification fatigue, configure Cluster-Auto-Recovery to consider a higher threshold for replication delays.

What is the maximum supported value for the replication delay threshold?
There is no limit as such.