TAS upgrade failing pre-start script on internal MySQL DB

Products

• VMware Tanzu Application Service

Issue/Introduction

TAS upgrade to 4.0.26 keep failing on GCP Central Prod02 Internal

Description: * 

TAS upgrade keep failing at mysql with below message 

Task 850361 | 14:07:23 | L starting jobs: mysql_proxy/### (0) (canary)
Task 850361 | 14:07:28 | Updating instance mysql: mysql/### (0) (canary) (00:03:23)
                      L Error: Action Failed get_task: Task 01d5a8e8-aeb0-433e-4334-563ae4aaaa18 result: 1 of 7 pre-start scripts failed. Failed Jobs: pxc-mysql. Successful Jobs: loggregator_agent, sysctl, bpm, bosh-dns, syslog_forwarder, dynatrace-oneagent.
Task 850361 | 14:07:54 | Updating instance mysql_proxy: mysql_proxy/#### (0) (canary)
...
Updating deployment:
 Expected task '850361' to succeed but state is 'error'
Exit code 1
Task 850361 | 14:12:04 | Error: Action Failed get_task: Task 01d5a8e8-aeb0-433e-4334-563ae4aaaa18 result: 1 of 7 pre-start scripts failed. Failed Jobs: pxc-mysql. Successful Jobs: loggregator_agent, sysctl, bpm, bosh-dns, syslog_forwarder, dynatrace-oneagent.
Task 850361 Started  Tue Sep 10 13:46:26 UTC 2024
Task 850361 Finished Tue Sep 10 14:12:04 UTC 2024
Task 850361 Duration 00:25:38
Task 850361 error

Additional symptom include when you ssh into the failing MySQL VM and using monit throws this error

mysql/2dbd17d6-39d2-46ea-8911-51ca2e39a64b:~$ sudo monit restart all
/var/vcap/bosh/etc/monitrc:8: Warning: include files not found '/var/vcap/monit/job/*.monitrc'

mysql/2dbd17d6-39d2-46ea-8911-51ca2e39a64b:~# monit summary
/var/vcap/bosh/etc/monitrc:8: Warning: include files not found '/var/vcap/monit/job/*.monitrc'
The Monit daemon 5.2.5 uptime: 1h 3m

System 'system_cdcbc9f3-62a8-4ae7-836a-aa5d5fdbfdd2' running

Also when you check vm state using bosh vms the MySQL is in a stopped state:

Environment

 

TAS 4.0.x

Cause

This is a known Percona bug due to this bug Pre start is failing due to corrupt tables:

One way to detect if there are corrupt tables is by makeing a back up, if it fails then there might be a corruption with the schema. Below is a log for successful backup:

 

 

# /var/vcap/packages/percona-xtrabackup-8.0/bin/xtrabackup --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf --backup --stream=xbstream > /dev/null ... 2024-09-11T18:43:04.672970-00:00 0 [Note] [MY-011825] [Xtrabackup] completed OK!

Resolution

There are 2 ways to resolve this issue:

• Restore from backup either using BBR or manually 
• If you don't have backup, please create a Support Request to engage R&D to help recreate the corrupted tables. Please note that this may result in a data loss. It is important to have a regular backup.