NSX-T Manager Backup task never completes

Products

• VMware NSX

Issue/Introduction

• NSX-T Manager backup task is running indefinitely. It never completes, nor fails
• On Backup server you will see that there have been no backups created for this duration confirming that the backup task is stuck
• Even if you reboot the Manager, as soon as the manager is up, it starts the backup task again
• Since a current backup task is running you are not able to edit/disable the backup configuration
• Checking on the /var/log/proton/nsxapi.log of the NSX-T Manager would reveal that the API calls related to backup are failing due to Authentication issues with backup server. ex: 
  
  

  2024-06-28T02:50:57.399Z ERROR http-nio-127.0.0.1-7440-exec-11 LocalBackupConfigurationFacadeImpl 135678 - [nsx@6876 comp="nsx-manager" errorCode="MP29107" level="ERROR" reqId="5cb3dc23-####-####-####-f7e85d3312bd" subcomp="manager" username="admin"] Error during read from SFTP server: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures
  
  or
  
  2024-06-28T07:23:23.212Z  INFO http-nio-127.0.0.1-7440-exec-8 NsxBaseRestController 171123 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/cluster/restore/backuptimestamps caused by exception com.vmware.nsx.management.common.exceptions.BaseException:  {"moduleName":"backup-restore","errorCode":29258,"errorMessage":"Error while trying to connect session or to get sftp channel to fileserver: 192.168.110.10, port: 22, user: root; Please see the documentation for supported host key algorithms.","details":"com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures "}

   

• If you try to ssh into the backup server using the same credentials configured on the NSX-T Manager backup settings, from NSX-T Manager root cli where the backup is failing, it fails, stating,

Permission denied, please try again

or

%ssh_dispatch_run_fatal: connection to 192.168.110.10 on port 22: Broken pipe

 

Environment

VMware NSX, VMware NSX-T

Cause

•  If the backup server credentials got changed but that is not updated on the NSX-T Manager side. (Permission denied, please try again)
• The ssh/sftp client on NSX-T Manager has stopped working properly (ssh_dispatch_run_fatal.*Broken pipe)

Resolution

To resolve the issue,

• Ensure the credentials from backup server end and the NSX-T Manager backup configuration end are same. If required,change the credentials on the backup server side to match with NSX-T Manager.
• Restart the following service on NSX-T Manager:

systemctl restart nsx-mp-api-server