Running mysql-diag returns "Client.Timeout when checking canary status"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When running a mysql-diag on the MySQL for VMware Tanzu monitor, VM returns the following as part of its output:

Checking canary status… Checking canary status... Get http://###.#.#.#:####/api/v1/status: net/http request canceled (Client.Timeout exceeded while awaiting headers)

Resolution

Checking the MySQL monitor/replication-canary.stderr.log shows the following:

http: Accept error: accept tcp 0.0.0.0:####: accept4: too many open files; retrying in 1s

Port 8111 is the port on which the replication-canary process listens.

You can verify this by running the command:

lsof -i:8111

This error indicates that the number of open files on the VM has exceeded its limit.

We can verify this by running the following from the monitor VM:

1. netstat -anp | grep LISTEN | grep tcp | grep replication. This should give you an output as follows:

tcp 0 0 0.0.0.0:8111 0.0.0.0:* LISTEN <NUMBER>/replication-ca

2. Taking note of the value of <NUMBER>, run the command cat  /proc/<NUMBER>limits | grep files. This should give you an output as follows:

Max open files            1024                 4096                 files 

3. In the above example, 1024 is the "Max open files" Soft Limit, and 4096 is the Hard Limit.

4. Finally, run cd /proc/<NUMBER>/fd and run the command ls -l | wc -l. The output should be higher than the "Max open files" Soft Limit, and is the reason for the error. This can be resolved with a bosh restart of the MySQL monitor VM.