SFTP Backup Registration Fails with SSH Fingerprint Error on SDDC Manager

Products

• VCF Operations

Issue/Introduction

When attempting to register or add a Linux-based SFTP backup server in SDDC Manager, the task fails during validation or registration. This issue is typically encountered during initial VCF 9.0+ deployments or when reconfiguring backups in hardened VCF 5.x environments. Users may see misleading errors regarding directory permissions while the underlying failure is related to cached host key mismatches.

• The task fails with the error: unable to make directory on the backup server.
• The API call used for registration fails with: unable to fetch the SSH fingerprint.
• In the log directory /var/log/vmware/vcf/operationsmanager/operationsmanager.log of SDDC manager, following error is seen:

YYYY-MM-DD HH:MM:SS 153+0000 DEBUG [vcf_om,69eb8beaaee4935] [c.v.e.sddc.common.util.SshKeyHelper,pool-2-thread-3] Found ecdsa SSH key for host <Backup-FQDN>: {"host":Backup_FQDN","keyType":"ecdsa-sha2nistp256","key":"[KEY_STRING]"}
YYYY-MM-DD HH:MM:SS.153+0000 INFO  [vcf_om,69eb8beaaee4935 [c.v.v.c.f.p.b.r.v.BackupLocationValidator,pool-2-thread-3] Given SSH fingerprint didn't match any of the fingerprints on server Backup_FQDN

Environment

VMware Cloud Foundation 5.x

VMware Cloud Foundation 9.x

Cause

• Unsupported Key Type: By design, VMware Cloud Foundation (VCF) 9.0 and higher require an ECDSA-based SSH fingerprint for SFTP backup configuration. RSA keys are not supported for fingerprint authentication in this release.
• Incorrect File Permissions: The known_hosts files on the SDDC Manager have incorrect permissions or ownership, preventing the system from writing or updating the fingerprints even after a successful manual injection attempt.
• Fingerprint Mismatch: If the SFTP server was re-imaged or its keys regenerated, the SDDC Manager’s cached entry in its internal trust store will reject the session.

Resolution

Ensure the backup server provides an ECDSA key and correct the permissions for the relevant known_hosts files on the SDDC Manager.

Step 1: Correct File Permissions and Ownership

Log in to the SDDC Manager via SSH as root and execute the following commands to set the correct permissions (644) and ownership:

1. Root SSH: chmod 644 /root/.ssh/known_hosts && chown root:root /root/.ssh/known_hosts
2. Common Services: chmod 644 /etc/vmware/vcf/commonsvcs/known_hosts && chown vcf_commonsvcs:vcf /etc/vmware/vcf/commonsvcs/known_hosts
3. VCF User: chmod 644 /home/vcf/.ssh/known_hosts && chown vcf:vcf /home/vcf/.ssh/known_hosts
4. Default Hosts: chmod 644 /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts && chown vcf_commonsvcs:vcf /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts

Step 2: Refresh and manually inject ECDSA Key

1. Refresh internal list: curl -k -X POST http://localhost/appliancemanager/ssh/knownHosts/refresh
2. Retrieve the key from the backup server: ssh-keyscan -t ecdsa [SFTP_SERVER_FQDN]
3. Inject the key

curl -i -X POST 'http://localhost:7100/appliancemanager/ssh/knownHosts' -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"knownHosts":[{"host":"[SFTP_SERVER_FQDN]","port":22,"keyType":"ecdsa-sha2-nistp256","key":"[KEY_STRING]"}]}'

Note: Replace [SFTP_SERVER_FQDN] with backup server address and [KEY_STRING] with the string retrieved in Step 2 point 2.

Step 3: Register via API (JSON Payload)

1. Use the ecdsa key-string  from step 2 command point 2: ssh-keyscan -t ecdsa [SFTP_SERVER_FQDN] 
2. Use  below JSON on  SDDC Manager to inject the ECDSA-based key into the system's known hosts.

{ "knownHosts": [ { "host": "[SFTP_SERVER_FQDN]", "port": 22, "keyType": "ecdsa-sha2-nistp256", "key": "[KEY_STRING]" } ]}