Due to bug in the MySQL v3.3.0 tile, the backup processes are not terminated and hang indefinitely until adbr-agent is restarted, or the MySQL restarts.

Products

• VMware Tanzu MySQL

Issue/Introduction

Due to bug in the MySQL v3.3.0 tile, the backup processes are not terminated and hang indefinitely until adbr-agent is restarted, or the MySQL restarts.This backup bug manifests when backups fail and we do see there is some configuration error in this environment:

[ERROR] 2024/11/21 08:00:09 backup.go:105: failed to upload artifact: Get "https:<dns-name>//storage/v1/b?alt=json&pageToken=&prefix=&prettyPrint=false&project=pcf-np-cah&projection=full": oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_grant","error_description":"Invalid JWT Signature."}

This is likely an issue with the Backup blobstore configuration, invalid credentials seem to have been provided and so backups cannot be uploaded to the blobstore. This failure causes the backup to fail.  Due to a bug in the MySQL v3.3.0 tile, the backup processes are not terminated and hang indefinitely until adbr-agent is restarted, or the MySQL restarts. In particular the backup utility xtrabackup is using about 130MiB for each failed backup, and over time this leads to excess memory on this system among other problems.

https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-for-mysql-on-cloud-foundry/3-3.html

Resolution

For short term fix:

Fixing the backup configuration so that backups do not fail,  Currently every backup is failing due to a configuration issue
Changing the backup schedule so it runs less frequently or not at all based on requirements on this environment
Restarting the `adbr-agent` process on the MySQL VMs to clear out failed backup processes using restart adbr-agent
 

If backups are not desired for whatever reason, changing the backup schedule to say, yearly or some similarly infrequent or invalid schedule can prevent backups from running and also sidestep this bug. See the Tanzu MySQL docs About configuring automated backups.

 

For permanent fix:

Upgrade to mysql v3.3.1, where we included a fix for this bug.

Bugfix: Backup process can exit successfully during network failures and when the backup is configured incorrectly in Tanzu Operations Manager

 

https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-for-mysql-on-cloud-foundry/3-3/mysql-for-tpcf/backup-restore.html