vCenter vami backups to storage cluster fail with error "deprecated key derivation used"

Products

• VMware vCenter Server

Issue/Introduction

When the VCSA backup target is a load-balanced cluster of SFTP servers, the backup may fail due to StrictHostKeyChecking. Because each node in the cluster may present a different SSH host key, the VCSA rejects the connection when the key does not match the entry in its trusted hosts file.

This issue often surfaces after an upgrade to 8.x, as the backup engine has transitioned to a more restrictive security model for file-based backups.

Symptoms:

• Backup job fails at the "Backup Lotus DB" or "Backup Stellar DB" stage.

• Log file /var/log/vmware/applmgmt/backup.log contains the following patterns:

  • ERROR: Failed to backup Lotus DB

  • ERROR: Failed to backup Stellar DB

  • stderr: *** WARNING : deprecated key derivation used.

  • stderr: error writing output file

  • util.Common.BackupRestoreError: Failed to backup Lotus DB

Environment

 

• Product: VMware vCenter Server Appliance (VCSA)

• Versions: 7.x, 8.x, 9.x

 

Cause

The VAMI backup service does not use the standard /root/.ssh/known_hosts file. Instead, it uses /etc/applmgmt/appliance/br_known_hosts. Furthermore, the service is hardcoded to use StrictHostKeyChecking=yes. If the storage cluster uses multiple nodes behind a single VIP/FQDN, the VCSA will fail the connection as soon as it hits a node whose specific host key is not already cached in the br_known_hosts file.

The error error writing output file typically signifies a failure in the SSH/SCP handshake or a lack of write permissions/trust on the target destination.

The hard failure is caused by the VAMI backup engine's inability to authenticate or verify the host key of the backup server using the specialized br_known_hosts file.

Resolution

To resolve the write failure, manually establish the SSH trust relationship between the VCSA and the backup target.

1. SSH to VCSA as root.

2. Scan and append the host keys of the backup server (including all nodes if using a cluster/VIP) to the appliance-specific trust file:

   Bash

    

   ssh-keyscan -t rsa <Backup-Server-IP-or-FQDN> >> /etc/applmgmt/appliance/br_known_hosts
   

3. Set correct ownership and permissions to allow the applmgmt service to access the keys:

   Bash

    

   chown applmgmt:applmgmt /etc/applmgmt/appliance/br_known_hosts
   chmod 600 /etc/applmgmt/appliance/br_known_hosts
   

4. Restart the Backup operation from the VAMI (Port 5480).

Additional Information

The br_known_hosts file is distinct from /root/.ssh/known_hosts. Even if a manual SSH connection from the command line works, the VAMI backup process will fail unless the keys exist in the /etc/applmgmt/appliance/ directory.