NSX-T Manager Backup task never completes

Products

• VMware NSX

Issue/Introduction

• NSX Manager backup task is running indefinitely. It never completes, nor fails
• On the Backup server, there have been no backups created for this duration, confirming that the backup task is stuck
• Even if you reboot the Manager appliance, as soon as the manager is up, it starts the backup task again
• Editing or disabling the backup configuration is not available since there is a current backup task running 
• Checking on the /var/log/proton/nsxapi.log of the NSX-T Manager reveals that API calls related to backups are failing due to authentication issues with the backup server. Examples: 
  

  2024-06-28T02:50:57.399Z ERROR http-nio-127.0.0.1-7440-exec-11 LocalBackupConfigurationFacadeImpl 135678 - [nsx@6876 comp="nsx-manager" errorCode="MP29107" level="ERROR" reqId="5cb3dc23-####-####-####-f7e85d3312bd" subcomp="manager" username="admin"] Error during read from SFTP server: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures
  
  or
  
  2024-06-28T07:23:23.212Z  INFO http-nio-127.0.0.1-7440-exec-8 NsxBaseRestController 171123 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/cluster/restore/backuptimestamps caused by exception com.vmware.nsx.management.common.exceptions.BaseException:  {"moduleName":"backup-restore","errorCode":29258,"errorMessage":"Error while trying to connect session or to get sftp channel to fileserver: 192.168.110.10, port: 22, user: root; Please see the documentation for supported host key algorithms.","details":"com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures "}

   

• Connecting by SSH from the NSX Manager (root) to the backup server, using the same credentials configured on the NSX Manager backup settings, fails with either:

Permission denied, please try again

or

%ssh_dispatch_run_fatal: connection to 192.168.110.10 on port 22: Broken pipe

 

Environment

VMware NSX 4.1.x and lower 
VMware NSX-T Datacenter

Cause

• If the backup server credentials was changed but was updated on the NSX-T Manager side. (Permission denied, please try again)
• If the SSH/SFTP client on NSX Manager has stopped working properly (ssh_dispatch_run_fatal.*Broken pipe)

Resolution

To resolve the issue,

• Ensure the credentials from backup server end and the NSX-T Manager backup configuration end are the same. If required, change the credentials on the backup server to match with NSX Manager.
• Restart the following service on NSX Manager:

systemctl restart nsx-mp-api-server

Starting NSX v4.2.0, backup and restore UI is enhanced to return authentication error and task ends. This enables user to retry backup with updated password / configuration