BBR backups of TAS using an external GCP database fails with "Failed to verify the server certificate via X509 certificate matching functions"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When using TAS that is configured with an external GCP database, backing up the foundation using BBR fails with the following error:

Error attempting to run backup for job bbr-credhubdb on backup_restore/3d1c33ec-ad47-44c9-8826-1fc3c8629cb4: + JOB_PATH=/var/vcap/jobs/bbr-credhubdb
+ SDK_PATH=/var/vcap/jobs/database-backup-restorer
+ BBR_ARTIFACT_FILE_PATH=/var/vcap/store/bbr-backup/bbr-credhubdb//credhubdb_dump
+ CONFIG_PATH=/var/vcap/jobs/bbr-credhubdb/config/bbr.json
+ /var/vcap/jobs/database-backup-restorer/bin/backup --config /var/vcap/jobs/bbr-credhubdb/config/bbr.json --artifact-file /var/vcap/store/bbr-backup/bbr-credhubdb//credhubdb_dump
ERROR 2026 (HY000): SSL connection error: Failed to verify the server certificate via X509 certificate matching functions
2021/04/30 19:03:16 ERROR 2026 (HY000): SSL connection error: Failed to verify the server certificate via X509 certificate matching functions - exit code 1

Environment

Product Version: 2.9

Resolution

If your deployment uses a GCP or Azure external database for TAS for VMs that is TLS-enabled, you must deselect the Enable hostname validation checkbox. For more information, see Disable Hostname Validation for External Databases on GCP and Azure.


You also need to check the bosh disable hostname verification in the CredHub tile:


This needs to be checked because the certs used on GCP and Azure databases do not have a hostname and/or IP has the Common Name:


The expected behavior is that the bbr.json file created during the backup would honor the tile selections and add an entry of tls.skip_host_verify: true to the file, but it does not. Therefore, during the BBR and when backing the external databases through TLS, we attempt to verify the hostname in the cert but there isn't one so the backup fails. 

Permanent fix:

Currently, there is no permanent fix for the issue, but we are working on getting this patched soon. 

Workaround:

The workaround for this issue is to manually edit the bbr.json file and add the flag "skip_host_verify": true to it.

To accomplish this do the following:

1. ssh to your OpsMan VM and authenticate with BOSH in order to run bosh commands.

2. ssh to the VM named backup_restore/xxxx within the TAS deployment

• bosh -d <cf-xyz> ssh backup_restore/xyz

3. Change to root in the VM:

• sudo -i

4. vim the bbr.json file:

• vim /var/vcap/jobs/bbr-credhubdb/config/bbr.json

5. Add the flag "skip_host_verify": true to the tls section:

"tls": {
  "skip_host_verify": true,
  "cert": {
    "ca": "----CERT----"
  }
}

After this change, you should be able to backup your TAS deployment. Keep in mind that any recreation / upgrade of the backup_restore VM will revert the values back so you will have to make the edits again.