vCenter Server VAMI backups to a load-balanced storage cluster fail with error "deprecated key derivation used"

Products

• VMware vCenter Server

Issue/Introduction

When configuring the vCenter Server Appliance (VCSA) backup target to a load-balanced cluster of SFTP servers, the backup process may fail due to StrictHostKeyChecking policies. Because each node in the storage cluster may present a different SSH host key, the VCSA rejects the connection when the presented key does not match an entry in its trusted hosts file. This issue is frequently observed after upgrading to vCenter 8.x, as the backup engine transitioned to a more restrictive security model for file-based backups.

Symptoms:

• Backup job fails at the "Backup Lotus DB" or "Backup Stellar DB" stage.

• Log file /var/log/vmware/applmgmt/backup.log contains the following patterns:

ERROR: Failed to backup Lotus DB
ERROR: Failed to backup Stellar DB
stderr: *** WARNING : deprecated key derivation used.
stderr: error writing output file
util.Common.BackupRestoreError: Failed to backup Lotus DB

 

Environment

VMware vCenter Server Appliance (VCSA) 7.x

VMware vCenter Server Appliance (VCSA) 8.x

VMware vCenter Server Appliance (VCSA) 9.x

 

Cause

The VAMI backup service does not use the standard /root/.ssh/known_hosts file. Instead, it uses /etc/applmgmt/appliance/br_known_hosts. Furthermore, the service is hardcoded to use StrictHostKeyChecking=yes. If the storage cluster uses multiple nodes behind a single VIP/FQDN, the VCSA will fail the connection as soon as it hits a node whose specific host key is not already cached in the br_known_hosts file.

The error error writing output file typically signifies a failure in the SSH/SCP handshake or a lack of write permissions/trust on the target destination.

The hard failure is caused by the VAMI backup engine's inability to authenticate or verify the host key of the backup server using the specialized br_known_hosts file.

Resolution

To resolve the write failure, manually establish the SSH trust relationship between the VCSA and the backup target.

1. SSH to VCSA as root.

2. Scan and append the host keys of the backup server (including all nodes if using a cluster/VIP) to the appliance-specific trust file:

   ssh-keyscan -t rsa <Backup-Server-IP-or-FQDN> >> /etc/applmgmt/appliance/br_known_hosts
   

3. Set correct ownership and permissions to allow the applmgmt service to access the keys:

   chown applmgmt:applmgmt /etc/applmgmt/appliance/br_known_hosts
   chmod 600 /etc/applmgmt/appliance/br_known_hosts
   

4. Restart the Backup operation from the VAMI (Port 5480).

Additional Information

The br_known_hosts file is distinct from /root/.ssh/known_hosts. Even if a manual SSH connection from the command line works, the VAMI backup process will fail unless the keys exist in the /etc/applmgmt/appliance/ directory.

Post implementing above resolution, incase, it still fails with the below log pattern in var/log/vmware/applmgmt/backup.log:

1][Failed to dispatch full database files.

*** WARNING : deprecated key derivation used.

Using -iter or -pbkdf2 would be better.

error writing output file

Please go through the KB article:  vCenter Server 9.0.x scheduled file-based backups fail when using 'Retain Last X Backups' policy