Error: Quorum loss during add-node operation in two-node cluster, Privileged Access Manager
search cancel

Error: Quorum loss during add-node operation in two-node cluster, Privileged Access Manager

book

Article ID: 444991

calendar_today

Updated On:

Products

CA Privileged Access Manager (PAM)

Issue/Introduction

After taking out a node from a 3-node cluster and adding it back, cluster becomes unstable with the following symptoms:

The node just added, node A shows up as out of sync and replication is missing

  • One of the two nodes making the cluster, node B, becomes unreachable, with messages about connection refused, even if in cluster manager it is showing as being in sync and replication to it is also showing fine
  • The third node in the cluster, node C is showing as up also with replication present

The present article describes how such a situation may occur for a working three node cluster

Environment

PAM 4.2.x 

2 node cluster

Cause

The operation may fail due to a combination of connection timeouts and ProxySQL state management logic.

  • Initial Connection Failure: the request for node A to join the cluster begins to be processed, but then, while this is happening there ia a connection failure between node A and node B. This causes an issue because of the following: when a new node joins a primary site cluster, ProxySQL adds it to the writer group. After determining the GTID, it removes appropriate nodes to maintain the policy of having only one writer group. In this case, however, since node B could not establish a communication to node A, ProxySQL retries three times and, if not successful, marks node A as SHUNNED

 

  • The ProxySQL logic triggers and automatic failover only when a node reaches the OFFLINE state, which will only happen if there is a successful connection that later fails at the SQL query level. This does not happen in this cae as the connection between node A and node B has never been established due to communication issues, which means that node A stays in a SHUNNED loop and the writer group becomes empty and hence it is not possible to service database requests.


Unfortunately, continuously waiting for node to became available to move further is a default requirement for proxysql, so there us no control on this other than meeting the requirements of having nodes reachable, which does not happen

Following errors will be seen in the MySQL logs

2026-05-14 07:32:55 MySQL_Session.cpp:2823:handler_again___status_CONNECTING_SERVER(): [ERROR] Max connect timeout reached while reaching hostgroup 2 after 10000ms . HG status: [{"srv_host":"<node_B>","status":"OFFLINE_HARD"},{"srv_host":"<node_A>","status":"SHUNNED"}]

2026-05-14 07:34:54 MySrvC.cpp:127:connect_error(): [ERROR] Shunning server <node_A>:3307 with 5 errors/sec. Shunning for 10 seconds
2026-05-14 07:34:54 MySQL_Session.cpp:2823:handler_again___status_CONNECTING_SERVER(): [ERROR] Max connect timeout reached while reaching hostgroup 2 after 11362ms . HG status:

  • The situation will continue until a new primary is elected. In the meantime, and until a new majority is established, node B will enter a quorum loss mode situation. If node B and node C are able to establish a quorum, the situation for the cluster may remediate itself. The quorum loss situation may occur node C restarts as part of the add-node procedure: when this happens, node B, the only remaining, may enter quorum loss mode and have issues confirming the ProxySQL writer topology  until node A is able to join the cluster.

 

  • While there is this ongoing situation there may be as well messages about "self-signed certificate in certificate chain" for node A. This is due to it being still in the mid-bring-up phase and not being yet ready for communication.


2026-05-14 07:34:56 mysql_connection.cpp:1238:handler(): [ERROR] Failed to mysql_real_connect() on 2:<node_A>:3307 ... 2026: TLS/SSL error: self-signed certificate in certificate chain.

2026-05-14 07:35:22 mysql_connection.cpp:1238:handler(): [ERROR] Failed to mysql_real_connect() on 2:<node_A>:3307 , FD (Conn:803 , MyDS:803) , 2026: TLS/SSL error: self-signed certificate in certificate chain.

2026-05-14 08:51:31 mysql_connection.cpp:1238:handler(): [ERROR] Failed to mysql_real_connect() on 2:<node_A>:3307 , FD (Conn:170 , MyDS:170) , 2026: TLS/SSL error: self-signed certificate in certificate chain.


Node addition is a common process and we don't expect any issues, in this case the root cause is connectivity failure for a prolonged time caused primary node to left in a state where there were no writer available to serve the requests. When joining a cluster, the joining node (e.g. nopde A) must stop Tomcat (web/CSPM) and the Credential Manager to reload MySQL and clone the master database. Temporary interruption on the joining node is normal, also reload of mysql groupreplication data by proxysql, there is not service down here just a reload.

Please note that there is a configurable safety timer (GROUP_REPLICATION_QUORUM_TIMEOUT) designed to prevent over-reaction to transient blips. If the writer group does no recover within 5 minutes, as is the case here, it will enter a quorum loss situation.

Resolution

This is a rare situation which has occurred because of the combination of the factors outlined in the Cause section, and therefore it should not be a usual situation

The following best practices are recommended

  • Always take a maintenance window before performing any cluster related operations
  • First click on "Save config locally" and then "save to cluster" after cluster site changes and then click on subscribe to active cluster
  • Make sure the joining server has same certificates as of cluster
  • Enable webservice log level to debug on primary master to capture more detailed information in any case of failures
  • Always have debug patch installed to troubleshoot any issues
Powered by Wolken Software