Backup Configuration fails during Backup Location validation

Products

• VMware SDDC Manager

Issue/Introduction

Unable to configure VCF backup(unable to register a backup server with VCF) in the "Site Settings" section of the "Backup" page in the SDDC-Manager UI, receiving the below error:

Invalid parameter: Validation failed for directory path <directory-path> on server <server-ip-or-fqdn>. Please make sure backup directory is intact and sftp server has write permission on backup path.

You can find the error below in /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

2024-05-07T14:05:51.082+0000 ERROR [vcf_om,ef7e96f5cde4418d,871a] [c.v.evo.sddc.common.util.SshUtil,pool-2-thread-18] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: reject HostKey: 10.10.10.10

Environment

VCF 4.x
VCF 5.0
VCF 5.1.0

Cause

When the backup server uses OpenSsh greater than or equal to 8.8, by default the server would not support the ssh-rsa SHA1 host key algorithm(for host key verification) but it is required in the Backup-Configuration validation API as it always looks for ssh-rsa host key for the host key verification.

Resolution

This issue has been fixed in VCF 5.1.1 and later releases.

To workaround the issue:

Add the backup server's ECDSA key to the SDDC-Manager, and then use the API to configure VCF backup.

1. Add the backup server's ECDSA key to the known_hosts of SDDC-Manager by following the below steps.
   1. SSH into the SDDC-Manager and switch to the root user.
   2. Fetch the ECDSA key of the backup server by running the command:
      "ssh-keyscan -t ecdsa -p <port> <backup-server-ip-or-fqdn> 2>/dev/null"
      
      
   3. Add the above key to the known-hosts of the SDDC-Manager by running the command:
      curl 'http://localhost/appliancemanager/ssh/knownHosts' -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"knownHosts":[{"host":"<backup-server-ip-or-fqdn>","port":22,"keyType":"ecdsa-sha2-nistp256","key":"<key-from-previous-step>"}]}'
      
      
2. Fetch ssh-rsa SHA256 fingerprint of the backup server(this fingerprint will be used in the next step to registering the backup server with VCF) using the command:
   "ssh-keygen -lf <(ssh-keyscan -t ssh-rsa -p 22 <ip-or-fqdn-of-the-server> 2>/dev/null)"
   
   
3. Register the backup server with VCF using API.
   1. Login into SDDC-Manager UI and go to "Developer Center" -> "API Explorer" -> "APIs for managing Backups and Restore" -> "PATCH /v1/system/backup-configuration"
   2. Use this JSON as the API body(populate the required fields):
      {"backupLocations":[{"server":"<backup-server-ip-or-fqdn>","port":22,"protocol":"SFTP","directoryPath":"<directory-path>","sshFingerprint":"SHA256:oxNDD6x/ZJ5MKKGjpF5/bvk3ZGfLOJEg3oySImzbjHM","username":"<user>","password":"<password>"}],"encryption":{"passphrase":"<passphrase>"}}
      
      
   3. Execute the API and wait until the Backup-Configuration task to completed successfully(can track the task status on the Tasks panel).
4. Ensure that the SDDC-Manager backups are working fine by triggering a backup using SDDC-Manager UI or API.