Upgrade Tanzu for VMs from 4.0.24 to 6.0.12 - internal database failure

Products

• VMware Tanzu Application Service

Issue/Introduction

• Upgrading from Tanzu for VMs 4.0.24 to 6.0.12, users might see an internal database failure.
• The mysql-metrics process will fail to start on the first canary instance during the Bosh Apply Changes with errors like (on the internal mysql DB VM located on the cf deployment):
  
  

  Task 12345 | 13:31:56 | Updating instance mysql: mysql/########-fdeb-4321-1234-############ (0) (canary) (00:07:02)

                         L Error: 'mysql/########-fdeb-4321-1234-############ (0)' is not running after update. Review logs for failed jobs: mysql-metrics

  Updating deployment:

    Expected task '12345' to succeed but state is 'error'

  Exit code 1
  
  

• From an SSH to the reported failed mysql instance, the /var/vcap/sys/log/mysql-metrics/mysql-metrics.stdout.log shows only the following messing (this is seen under normal conditions as well):
  
  

  {"timestamp":"2025-06-02T20:57:58.514571747Z","level":"error","source":"MetricsLogger","message":"MetricsLogger.error processing metrics","data":{"error":"No Previous Data"}}

Environment

This failure was seen when upgrading TAS from 4.0.24 to 6.0.12 when Dynatrace One Agent is installed.

Cause

The Dynatrace One agent might cause a delay in startup with the internal MySQL database's mysql-metrics process, preventing timely startup, resulting in timeouts. A similar issue is noted with Grafana jobs in the Healthwatch deployments in the this KB.

 

Refer to the Dynatrace to TAS compatibility noted here.

Resolution

The Dynatrace Oneagent needs to be excluded from deployment that it is causing to fail.

 

Perform the steps - 

1. Dump the current runtime config:
   

   bosh-runtime-config > runtime.cfg

2.  Edit the runtime config to exclude deployment from Dynatrace add-on
   

   $ vim runtime.cfg
   
   exclude:
     deployments:
        <failed deployment>

3. Update the runtime-config
   

   bosh update-runtime-config runtime.cfg  

4. Run a BOSH deploy or execute an apply changes in OpsManager.