"Lost connection to MySQL server during query" error in controller_ng after losing connection to MySQL server in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The Cloud Controller (CC) API is failing intermittently or inaccessible, and the service cloud_controller_ng is failing for the cloud_controller instance.

You may observe this error from the Cloud Controller logs in the file controller_ng:

{"timestamp":"2021-09-11T11:18:13.483778052Z","message":"2021-09-11T11:18:13+0000: [Worker(cc_api_worker.cloud_controller.0.2)] Error while reserving job: Mysql2::Error::ConnectionError: Lost connection to MySQL server during query","log_level":"info","source":"cc-worker","data":{},"thread_id":47005676312345,"fiber_id":47005734412123"process_id":6,"file":"/var/vcap/data/packages/cloud_controller_ng/53151e2e42324e72946a3184425e89e0d917fd29/gem_home/ruby/2.5.0/gems/delayed_job-4.1.5/lib/delayed/worker.rb","lineno":285,"method":"say"}

Environment

Product Version: 2.11

Resolution

Cloud Controller depends on the Tanzu Application Service for VMs (TAS for VMs) database availability.

When Bosh is updating/upgrading or recreating the TAS mysql proxy or TAS mysql server, this message will appear in the Cloud Controller logs.

If there was no recent Bosh update for TAS mysql proxy or TAS mysql server, you may review the health of the internal TAS for VMs MySQL cluster with a tool such as mysql-diag. For more information on mysql-diag, refer to Running mysql-diag.

Review the health of the external TAS for VMs database. For more information, refer to External System Database Configuration.