MariaDB cluster is partitioned

Products

• VMware Integrated OpenStack

Issue/Introduction

• The Galera cluster for MariaDB is partitioned and backups are failing
  "mariadb cluster is not in synced state"
• Health check shows similar:
   wsrep_cluster_size != 3   (3 is the default size, if your cluster has more nodes that would be the number listed)

Environment

7.x

Cause

This could be caused by a number of issues and would need to be investigated to determine cause and corrective action.

Resolution

Please contact Broadcom Support to investigate the issue.

Additional Information

We can run this command to see details about the Galera cluster for MariaDB.  Please capture this information and include it when you contact Broadcom Support.

for server in `seq 0 2`;do osctl exec -ti mariadb-server-$server -- mysql --defaults-file=/etc/mysql/admin_user.cnf --host=localhost -e "show status;";done |grep -e wsrep_cluster_size -e wsrep_last_committed -e wsrep_local_cached_downto -e wsrep_local_commits -e wsrep_local_state_comment -e wsrep_local_state_uuid -e wsrep_incoming_addresses

It will return results similar to below for each node.  The highest wsrep_last_committed number is the DB with the most current updates.

mariadb-server-0:

wsrep_cluster_size      1

wsrep_incoming_addresses        mariadb-server-0.mariadb-discovery.openstack.svc.viosea.local:3306

wsrep_last_committed    987654321

wsrep_local_cached_downto       123456789

wsrep_local_commits     1234567

wsrep_local_state_comment       Synced

wsrep_local_state_uuid  11111111-1111-1111-1111-111111111111

 

 

We can manually backup the DB with the following. Use the DB node with the highest wsrep_last_committed.  This would be recommended to do prior to contacting Broadcom Support.

example:

osctl exec -ti mariadb-server-0 -- mysqldump --defaults-file=/etc/mysql/admin_user.cnf --host=localhost --all-databases > /tmp/7.2.sql

(We added "--host=localhost" to make sure the backup comes from mariadb-server-0)