SDDC Manager Backup Configuration Failure: "Validation failed for directory path <path> on Ubuntu server <server ip/fqdn>. Please make sure backup directory is intact and sftp server has write permissions on backup path.

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• When attempting to configure an Ubuntu OS based SFTP server, you may see this Error:
• Error in /var/log/vmware/vcf/operationsmanager.log

  "Permanently added '<fqdn/ip>' (ECDSA) to the list of known hosts….. <fqdn>: Permission denied…
  ..........
  Permanently added '<fqdn/ip>' (RSA) to the list of known hosts….. <fqdn>: Permission denied…
  .........
  
  Error: jsch CLI session: 
  com.jcraft.jsch.JSchException: Auth fail"
  ...........
  
  Unable to create jsch CLI sesssion:
  com.jcraft.jsch.JSchException: reject HostKey: <fqdn/ip>"

• Updating the host keys will NOT address this issue. Reference KB How to update the SSH host keys on the SDDC Manager
• This issue is also NOT fixed by bypassing the certificates. Reference KB  Backup configuration fails with the SSH related error message

Environment

VMware Cloud Foundation

Cause

The SDDC Manager is expecting RSA whereas Ubuntu is using ECDSA.

The SDDC Manager UI appears to be forcing RSA. Thus using an API as a workaround to use the API call to force ECDSA.

Resolution

There is Currently no Resolution. It is projected to be fixed in VCF 5.2.

Workaround:

1. Add the backup server's ECDSA key to the known_hosts of the SDDC Manager by following the below steps.
   1. SSH into the SDDC Manager as a root user.
   2. Fetch the ECDSA key of the backup server by running the command "ssh-keyscan -t ecdsa -p <port> <backup-server-ip-or-fqdn> 2>/dev/null"
   3. Add the above key to the known_hosts file of the SDDC-Manager by running the command:

      curl 'http://localhost/appliancemanager/ssh/knownHosts' -i -X POST -H 'Content-Type:application/json' -H 'Accept: application/json' -d '{"knownHosts":[{"host":"<backup-server-ip- fqdn>","port":22,"keyType":"ecdsa-sha2-nistp256","key":"<key-from-previous-step>"}]}'

2. Fetch ssh-rsa SHA256 fingerprint of the backup server(this fingerprint will be used in the next step to registering the backup server with VCF) using the command:

   ssh-keygen -lf <(ssh-keyscan -t ssh-rsa 22 <ip-or-fqdn-of-the-server> 2>/dev/null)

3. Register the backup server with VCF using API.
   1. Login into SDDC-Manager UI and go to "Developer Center" -> "API Explorer" -> "APIs for managing Backups and Restore" -> "PATCH /v1/system/backup-configuration"
   2. Use this JSON as the API body(populate the required fields):

      {"backupLocations":[{"server":"<backup-server-ip-or-fqdn>","port":22,"protocol":"SFTP","directoryPath":"<directory path>","sshFingerprint":"SHA256:oxND#######jHM","username":"<user>","password":"<password>"}],"encryption":{"passphrase":"<passphrase>"}}

   3. Execute the API and wait until the Backup-Configuration task to complete successfully (you can track the task status on the Tasks panel).
4. Ensure that the SDDC-Manager backups are working fine by triggering a backup using SDDC-Manager UI or API.

Additional Information