NSX-T Manager Backup task never completes
search cancel

NSX-T Manager Backup task never completes

book

Article ID: 371358

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX-T Manager backup task is running indefinitely. It never completes, nor fails
  • On Backup server you will see that there have been no backups created for this duration confirming that the backup task is stuck
  • Even if you reboot the Manager, as soon as the manager is up, it starts the backup task again
  • Since a current backup task is running you are not able to edit/disable the backup configuration
  • Checking on the /var/log/proton/nsxapi.log of the NSX-T Manager would reveal that the API calls related to backup are failing due to Authentication issues with backup server. ex: 

    2024-06-28T02:50:57.399Z ERROR http-nio-127.0.0.1-7440-exec-11 LocalBackupConfigurationFacadeImpl 135678 - [nsx@6876 comp="nsx-manager" errorCode="MP29107" level="ERROR" reqId="5cb3dc23-####-####-####-f7e85d3312bd" subcomp="manager" username="admin"] Error during read from SFTP server: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures
    
    or
    
    2024-06-28T07:23:23.212Z  INFO http-nio-127.0.0.1-7440-exec-8 NsxBaseRestController 171123 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/cluster/restore/backuptimestamps caused by exception com.vmware.nsx.management.common.exceptions.BaseException:  {"moduleName":"backup-restore","errorCode":29258,"errorMessage":"Error while trying to connect session or to get sftp channel to fileserver: 192.168.110.10, port: 22, user: root; Please see the documentation for supported host key algorithms.","details":"com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures "}

     

  • If you try to ssh into the backup server using the same credentials configured on the NSX-T Manager backup settings, from NSX-T Manager root cli where the backup is failing, it fails, stating,

Permission denied, please try again

or

%ssh_dispatch_run_fatal: connection to 192.168.110.10 on port 22: Broken pipe

 

Environment

VMware NSX, VMware NSX-T

Cause

  •  If the backup server credentials got changed but that is not updated on the NSX-T Manager side. (Permission denied, please try again)
  • The ssh/sftp client on NSX-T Manager has stopped working properly (ssh_dispatch_run_fatal.*Broken pipe)

Resolution

To resolve the issue,

  • Ensure the credentials from backup server end and the NSX-T Manager backup configuration end are same. If required,change the credentials on the backup server side to match with NSX-T Manager.
  • Restart the following service on NSX-T Manager:

systemctl restart nsx-mp-api-server