Error "VCFMS-PRECHECK-BACKUP-CONFIG-002" during SFTP backups following the VCF 9.0.2.0 to 9.1.0 upgrade

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Following the upgrade of the environment from VMware Cloud Foundation (VCF) 9.0.2.0 to 9.1.0, the pre-existing Windows-based SFTP backup configuration was removed.

• Subsequent automated and manual backup operations fail even after the backup location has been reconfigured.
  
  
• The multi-component-backup-prechecks pod in the vmsp-platform namespace reports the following error during SFTP prechecks:
  
  k logs -n vmsp-platform multi-component-backup-prechecks-#####-sftp-precheck-#########
  
  

  "issue": {
              "id": "com.vmware.vcfms.system.precheck.backup.name.BackupConfiguration",
              "type": "ERROR",
              "message": {
                "id": "com.vmware.vcfms.system.precheck.backup.FailedCopyTestFile",
                "localized": "Failed to copy test file to server: server=<SFTP-SERVER-FQDN>, error=Failed to create file system for destination \"sftp:<Backup-Path>\": NewFs: couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch;.. [VCFMS-PRECHECK-BACKUP-CONFIG-002]",
                "args": [
                  "<SFTP-SERVER-FQDN>",
                  "Failed to create file system for destination \"sftp:<Backup-Path>\": NewFs: couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch;"
                ],
                "additionalProperties": null,
                "default": "Failed to copy test file to server: server=<SFTP-SERVER-FQDN>, error=Failed to create file system for destination \"sftp:<Backup-Path>\": NewFs: couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch;.. [VCFMS-PRECHECK-BACKUP-CONFIG-002]"
              },
              "resolution": {
                "id": "com.vmware.vcfms.system.precheck.backup.FailedCopyTestFile.remediation",
                "localized": "Failure during copying of the test file can be caused by network connectivity or firewall issues, server out of space issues, quota issues etc. If possible, check for such issues and try again. If the failure persists, please contact support.",
                "args": [
                ],
                "additionalProperties": null,
                "default": "Failure during copying of the test file can be caused by network connectivity or firewall issues, server out of space issues, quota issues etc. If possible, check for such issues and try again. If the failure persists, please contact support."
              },
              "additionalProperties": null
            },
            "additionalProperties": null
          }

Environment

VCF Operations 9.1

Cause

The backup failure is caused by an SSH host key mismatch, resulting from a non-session-sticky load balancer that routes traffic across unsynchronized active and passive SFTP nodes.

Resolution

Standard VCF backup guidelines do not support Windows-based SFTP servers or multi-node load-balanced/HA SFTP topologies.
For VCF backup targets, deploying Linux-based OpenSSH servers is strongly recommended.

If you are using a Windows-based SFTP configuration, apply one of the following workarounds:

• Option 1 (Recommended for HA Topologies): 
  Synchronize the SSH host key pairs across all SFTP nodes behind the load balancer.
  Export the host key certificate or private key from the active node and import it into the passive node to ensure both nodes present identical host key fingerprints.
  Please consult your SFTP software team for assistance with this configuration.

• Option 2 (Bypass Load Balancer): 
  Modify the VCF Operations backup configuration to target the FQDN or IP address of the active SFTP node directly, rather than using the SFTP Load Balancer VIP.