Manually forcing a MySQL node to rejoin the HA cluster failed
Article ID: 445219
Updated On:
Products
Issue/Introduction
When following the procedure on "Manually forcing a MySQL node to rejoin the HA cluster", the step to run the pre-start script was failing.
The mysql.err.log file shows these lines and error:
2026-06-19T20:46:55.283033Z 0 [Note] [MY-000000] [Galera] Member 0.0 (mysql/0) requested state transfer from '*any*'. Selected 1.0 (mysql/1)(SYNCED) as donor.
2026-06-19T20:46:55.283082Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1
2026-06-19T20:46:55.649798Z 0 [ERROR] [MY-000000] [Galera] /var/vcap/data/compile/percona-xtradb-cluster-8.0/Percona-XtraDB-Cluster-8.0.44-35/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1360: Will never receive state. Need to abort.
The above logs show that the donor selected is "mysql/1". The instance name of "mysql/1" can be determined by logging in to mysql_monitor instance, and running "mysql-diag" command and reviewing the output.
In the mysql.err.log of the donor, the following lines were logged:
2026-06-19T20:46:55.296200Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.x.x.x:4444/xtrabackup_sst//1' --socket '/var/vcap/sys/run/pxc-mysql/mysqld.sock' --datadir '/var/vcap/store/pxc-mysql/' --basedir '/var/vcap/data/packages/percona-xtradb-cluster-8.0/9999/' --plugindir '/var/vcap/data/packages/percona-xtradb-cluster-8.0/9999/lib/plugin/' --defaults-file '/var/vcap/jobs/pxc-mysql/config/my.cnf' --defaults-group-suffix '_plugin' --mysqld-version '8.0.44-35' '' --gtid 'xxxx:9999' )
2026-06-19T20:46:55.641765Z 0 [Note] [MY-000000] [WSREP-SST] mktemp: failed to create directory via template '/var/vcap/data/pxc-mysql/tmp/pxc_sst_XXXX': Read-only file system
2026-06-19T20:46:55.642979Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1
2026-06-19T20:46:55.646433Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.x.x.x:4444/xtrabackup_sst//1' --socket '/var/vcap/sys/run/pxc-mysql/mysqld.sock' --datadir '/var/vcap/store/pxc-mysql/' --basedir '/var/vcap/data/packages/percona-xtradb-cluster-8.0/9999/' --plugindir '/var/vcap/data/packages/percona-xtradb-cluster-8.0/9999/lib/plugin/' --defaults-file '/var/vcap/jobs/pxc-mysql/config/my.cnf' --defaults-group-suffix '_plugin' --mysqld-version '8.0.44-35' '' --gtid 'xxxx:9999' : 1 (Operation not permitted)
2026-06-19T20:46:55.649337Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.x.x.x:4444/xtrabackup_sst//1' --socket '/var/vcap/sys/run/pxc-mysql/mysqld.sock' --datadir '/var/vcap/store/pxc-mysql/' --basedir '/var/vcap/data/packages/percona-xtradb-cluster-8.0/9999/' --plugindir '/var/vcap/data/packages/percona-xtradb-cluster-8.0/9999/lib/plugin/' --defaults-file '/var/vcap/jobs/pxc-mysql/config/my.cnf' --defaults-group-suffix '_plugin' --mysqld-version '8.0.44-35' '' --gtid 'xxxx:9999'
The above logs indicate that there was a problem with the data transfer, and it failed to create a directory due to a read-only file system.
Environment
Tanzu Application Service
Cause
The donor mysql instance was problematic. It appears that the file-system that mysql process was using, became a read-only filesystem hence it was failing to execute the data transfer to the joining node.
Resolution
Recreate the donor mysql instance. Run the following command to recreate the instance:
bosh -d $cf-deployment recreate mysql/$GUID
If the recreate fails due to errors indicating file-system issues, then you can try to shutdown the VM either by running the "shutdown" command or doing it from vCenter. Once shut down, run "bosh cck" to recover the particular mysql instance that was shut down.
Feedback
