MySQL high persistent disk usage

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

TAS MySQL cluster, gcache.page.NNNNNNN files under /var/vcap/store/pxc-mysql directory had been building up.
 
The MySQL cluster  persistent disk is filling with a bunch of "gcache.page.NNNNNN" files.We are still researching the root cause however it has been observed to occur during upgrade from 2.13.12 to 3.0.7 and also been seen following a bootstrap.

Environment

Product Version: 3.0

Resolution

Login to mysql node with highest persistent disk utilisation and check the details surrounding the disk usage to validate if this node is being impacted by this issue:
 

mysql/11cd95d1-0dce-4c14-af75-1a2a9d2d7e48:/var/vcap/sys/log# df -h
Filesystem   Size Used Avail Use% Mounted on

tmpfs      4.8G 497M 4.3G 11% /run
/dev/sda1    4.9G 2.6G 2.1G 55% /
tmpfs      24G   0  24G  0% /dev/shm
tmpfs      5.0M   0 5.0M  0% /run/lock
tmpfs      4.0M   0 4.0M  0% /sys/fs/cgroup
/dev/sdb2    80G 948M  75G  2% /var/vcap/data
tmpfs      16M 416K  16M  3% /var/vcap/data/sys/run
/dev/sdc1    739G 570G 132G 82% /var/vcap/store
tmpfs      4.8G   0 4.8G  0% /run/user/1002

mysql/11cd95d1-0dce-4c14-af75-1a2a9d2d7e48:/var/vcap/store# du -sch *
16K   lost+found
3.9G  mysql_audit_logs
566G  pxc-mysql
570G  total

mysql/11cd95d1-0dce-4c14-af75-1a2a9d2d7e48:/var/vcap/store#
mysql/11cd95d1-0dce-4c14-af75-1a2a9d2d7e48:/var/vcap/store/pxc-mysql# ls -lhtr | grep gcache.page | wc -l

4093

First we need make sure the cluster is synced and healthy.

Login into mysql_monitor and check the health status of cluster using `mysql-diag`. More details about this procedure can be found here (https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-platform-for-cloud-foundry/5-0/tpcf/mysql-diag.html)

If the cluster is healthy then stop the galera-init of the highest utilisation mysql node. This should stop the creation of the "gcache.page.NNNNNN" files.
 

monit stop galera-init

For backup move the gcache files to another device/directory that has capacity so that we free up space on the persistent disk.

Once the files are moved, start the galera-init.

monit start galera-init

Post starting the galera make sure the cluster status shows healthy by running mysql-diag again.

An additional check to make sure there was a clean startup can be done by checking the /var/vcap/sys/log/pxc-mysql/mysql.err.log.