Manual Mysql backup fails with too many open files

Products

• VMware Tanzu SQL

Issue/Introduction

You try to manually backup your MySQL service instance
 

cf adbr backup <mysql-service-instance-name>

you check the status and this fails
 

cf adbr get-status <mysql-service-instance-name>

To get more info access the service instance logs :

checking into adbr-agent logs you see error 

[ERROR] 2022/10/18 09:00:02 backup.go:89: failed to get backup artifact: failed to execute command: /var/vcap/jobs/streaming-mysql-backup-client/bin/client: fork/exec /var/vcap/jobs/streaming-mysql-backup-client/bin/client: too many open files stdout: stderr: [ERROR] 2022/10/18 10:00:06 backup.go:89: failed to get backup artifact: failed to execute command: /var/vcap/jobs/streaming-mysql-backup-client/bin/client: fork/exec /var/vcap/jobs/streaming-mysql-backup-client/bin/client: too many open files stdout: stderr: [ERROR] 2022/10/18 11:00:06 backup.go:89: failed to get backup artifact: failed to execute command: /var/vcap/jobs/streaming-mysql-backup-client/bin/client: fork/exec /var/vcap/jobs/streaming-mysql-backup-client/bin/client: too many open files 

As of date of writing MySQL for VMs version 2.10.9 and older versions have the default configured open files for streaming-mysql-backup-client is 1024.

Environment

Product Version: 2.10

Resolution

Here are steps for workaround for this issue:

1. SSH into your service instance
2. Change the limits.open_files in /var/vcap/jobs/adbr-agent/config/bpm.yml so we it will look similar to 

   # cat /var/vcap/jobs/mysql-restore/config/bpm.yml
   ---
   processes:
   - name: mysql-restore
     limits:
       open_files: 65536
   ...

   
    
3. restart adbr-agent for configuration change to take effect 

   monit restart adbr-agent

4. retry backup 

   cf adbr backup <mysql-service-instance-name>

5. check the status of the back up 

   cf adbr get-status <mysql-service-instance-name>