While performing a NSX SFTP backup, the remote server is not connecting because the fingerprint is invalid

Products

VMware NSX

Issue/Introduction

After upgrading the NSX manager or the backup solution you are unable to trigger a backup due to the similar errors:

Error: Configured fingerprint did not match fileserver ...   (Error code: 29004)

There will be log entries on the manager node similar to:

nvpapi/api_server.log

2024-08-08T08:00:43.034Z napi.root.node.file-store.utils WARNING REPEATS: 1 repeats in 10 sec: Rejecting the remote server because fingerprint is invalid, expected: sha###############/w, actual: sha###############
2024-08-08T08:00:43.034Z napi.root.node.backup_restore ERROR REPEATS: 1 repeats in 10 sec: Cluster backup file copy operation failed due to 400 Bad Request Content-Type: application/json

syslog

2024-08-07T08:00:43.389Z nsx.tmgt.comNSX 82406 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="WARNING"] REPEATS: 1 repeats in 8 sec: Rejecting the remote server because fingerprint is invalid, expected: sha####################################/w, actual: sha####################################
2024-08-07T08:00:43.390Z nsx.tmgt.com NSX 82406 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="ERROR" errorCode="NOD110"] REPEATS: 1 repeats in 8 sec: Cluster backup file copy operation failed due to 400 Bad Request#015#012Content-Type: application/json#015#012Content-Length: 104#015#012Vmw-Task-Id: cd4b0442-####-5905-ecad-##########_ce7a066b-####-4500-9e35-ef2c3f45e21d#015#012#015#012{"error_code": 36226, "error_message": "Invalid fingerprint specified.", "module_name": "node-services"}

proton/nsxapi.log

2024-08-08T08:00:33.837Z  INFO scheduling-worker-1 NapiBackupGenerationServiceImpl 142125 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Exception received from napi statusDetail {"error_code": 36226, "error_message": "Invalid fingerprint specified.", "module_name": "node-services"}
2024-08-08T08:00:33.837Z ERROR scheduling-worker-1 NapiBackupGenerationServiceImpl 142125 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP29256" level="ERROR" subcomp="manager"] org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "{"error_code": 36226, "error_message": "Invalid fingerprint specified.", "module_name": "node-services"}"
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:101) ~[?:?]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:168) ~[?:?]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[?:?]
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[?:?]
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:825) ~[?:?]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:783) ~[?:?]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717) ~[?:?]
        at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:474) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.callHelper(NapiBackupGenerationServiceImpl.java:424) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.callBackupHelper(NapiBackupGenerationServiceImpl.java:287) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.createAndUploadGemFireBackup(NapiBackupGenerationServiceImpl.java:754) ~[?:?]
        at com.vmware.nsx.management.backup.steps.BackupDbImpl.createAndUploadBackup(BackupDbImpl.java:24) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.BackupGenerationServiceImpl.createAndUploadFullClusterBackup(BackupGenerationServiceImpl.java:262) ~[?:?]
        at com.vmware.nsx.management.backup.jobs.ClusterNodeBackupJob.execute(ClusterNodeBackupJob.java:33) ~[?:?]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[?:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[?:?]
2024-08-08T08:00:33.837Z ERROR scheduling-worker-1 BackupGenerationServiceImpl 142125 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP29004" level="ERROR" subcomp="manager"] Cluster backup for NSX_UFO_BACKUP_RESTORE had error
2024-08-08T08:00:33.868Z ERROR scheduling-worker-1 ClusterNodeBackupJob 142125 - [nsx@6876 comp="nsx-manager" errorCode="MP29004" level="ERROR" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with ; BackupAsyncStatus [BackupStatus [status=BAD_FINGERPRINT, statusDetail=Invalid fingerprint specified., remoteUri=sftp://###.###.###:22/apps/nsxbackup/pod1/cluster-node-backups/4.1.2.3.0.23382420-#######-3942-####-####-6f0f32ff452/backup-2024-08-08T08_00_00UTC/cluster_backup-#######-3942-####-####-6f0f32ff452d-10.##.###.57-nsx-ufo-backup-restore.tar, errorCode=null, startTime=1723104003601, endTime=1723104033837]; responseBody=null].

Environment

VMware NSX-T Data Center 3.x

Cause

The cryptography(HostKeyAlgorithm) changes after an upgrade or a configuration change

Resolution

Consult your SFTP backup solution vendor for support.

NOTE: the SFTP backup solution needs to be (re)configured to use a supported key algorithm/size

For additional information see Troubleshooting NSX Backup and Restore Failures

Additional Information

Ways to test this issue:

When configuring NSX that has a bad private Key, there will be an error similar to:
```
Error: Error fetching fingerprint of fileserver Algorithm negotiation failed: Possibly there is no ECDSA support for public keys (Error code: 29259)
```
Refer to the Administration Guide
NSX supports RSA for SSH private key generation using key sizes 1024-bits, 2048-bits, and 4096-bits. 4096-bits is recommended. If the command output does not return a supported ECDSA key, you must configure the key on the backup server. Contact the OS vendor if you need guidance for that configuration
Use an ESXI host as the source for the backup
1. Create a folder for the backups on a datastore
2. "cd" into that folder and collect that path
  Note: you can use command "pwd" to get the path quicker
3. Add the information into the backup and trigger the backup