VCF Services Runtime 9.1 backup reports false failure in VCF Operations UI due to SSH fingerprint mismatch

Products

• VCF Operations

Issue/Introduction

In VMware Cloud Foundation (VCF) Operations 9.1, when attempting to perform a backup of VCF Services runtime components to a SFTP server, the overarching task reports as "Failed" in the VCF Operations UI. Despite the failure status in the UI, users will observe that the backup files are successfully written and present on the target SFTP server.

 

VCF Operations UI Error Messages:
When viewing the task details for Perform backup of components using SDDC lifecycle service and Backup vSphere components, the following errors are displayed:

Task with ID '<TASK_ID>' from SDDC lifecycle with ID '<SDDC_LCM_ID>' has completed with failure(s). Check the errors in the next sub-task(s) for more details.

Failed to create backups for VCF services runtime components '[<LIST_OF_COMPONENT_IDS>]'. Verify components are accessible and sufficient storage is available.

SSH fingerprint mismatch detected for backup destination <SFTP_SERVER_FQDN>.

Environment

• VCF Operations 9.1
• VCF Services Runtime 9.1

Cause

This issue is caused by a false-failure reporting bug within the VCF Fleet LCM component (specifically BackupVspComponentsAction), combined with how SSH host keys are negotiated. VCF 9.1 management component backups are executed by individual microservice pods. When the target SFTP server is configured to offer multiple SSH host key algorithms, a temporary negotiation mismatch can occur. The VCF Backup Location configuration only accepts a single SSH fingerprint.

 

During the backup workflow, some pods experience a brief, transient SSH host-key negotiation warning when connecting to the server. The pods automatically retry, resolve the connection, and successfully transfer the backup data. However, these transient warnings are recorded as ERROR-level messages in the cumulative task history.

The bug in the VCF Fleet LCM logic causes it to treat any historical ERROR-level message generated during the run as a hard failure, which incorrectly overrides the final "SUCCEEDED" status of the actual backup operations.

 

The VCF Fleet LCM cumulative messages array logs the transient errors mid-run, despite the final status succeeding:

{"timestamp":##########,"stage":"component-backup","level":"ERROR","id":"com.vmware.vcfms.backup.SSHFingerprintMismatch",...}
{"timestamp":##########,"stage":"component-backup","level":"ERROR","id":"com.vmware.vcfms.backup.SSHFingerprintMismatch",...}
{"timestamp":##########,"stage":"component-backup","level":"ERROR","id":"com.vmware.vcfms.backup.SSHFingerprintMismatch",...}

Resolution

This is a known issue; the backups are completing successfully. Validate the same by logging directly into the SFTP server.

 

This issue will be fixed in a future release.

Workaround:
If the target SFTP server can be modified, users can eliminate the transient errors by restricting the SFTP server to offer only a single SSH host key algorithm that matches the fingerprint configured in the VCF Operations UI. If the SFTP server configuration cannot be modified, users can safely ignore the failure status in the VCF UI and rely on direct verification of the backup files on the target SFTP server.