Scheduled NSX backup fails with error "Either backup server login failed or unauthorized access to backup directory"

Products

• VMware NSX

Issue/Introduction

• Last Backup Status shows "Failed", however node and cluster Backup status is Successful.
  
• Manual Backup completes successfully.
• In nsxapi logs we see error code similar to the below observed:
  
  Log Location on manager: /var/log/proton/nsxapi.log
[nsx@ comp="global-manager" errorCode="MP29115" level="ERROR" s2comp="backup-restore" subcomp="global-manager"] Cluster backup failed with ; BackupAsyncStatus [BackupStatus [status=AUTH_FAILURE, statusDetail=Cannot connect to sftp server. The remote server returned error: status: 255   out: '[Errno 5] Input/output error'.
  
  
• SSHD logs collected from the Windows SFTP server indicates that connection between NSX manager and the SFTP server was closed due to MAXStartups.

Environment

VMware NSX

Cause

From the Windows SSHD logs we observed:

• The SSH/SFTP connection to the NSX Manager was terminated because the current volume of unauthenticated sessions surpassed the MaxStartups threshold specified in the SSH configuration.
• Consequently, the server then rejected new access attempts as it had reached the limit for concurrent connections awaiting authentication.

Resolution

Given the symptoms identified, it is recommended to engage the team responsible for managing your SFTP server to further investigate and troubleshoot the connection limitations on their end.

Additional Information

Following steps can be followed to collect SSHD logs from the Windows SFTP server:

• Enable sshd_config logging on windows backup server. Steps to enable sshd logging are as below:
  1a. Navigate to : C->ProgramData->ssh 
  1b. Look for file sshd_config 
  1c. Take a backup of the sshd_config, rename and save it was bak_sshd_config  
  1d. Edit the sshd_config file as follows:
• Open the Config File: Open C:\ProgramData\ssh\sshd_config using a text editor (like Notepad) as an administrator.
• Enable File Logging: Locate the #Logging section and add or modify the following lines:
  text
  SyslogFacility LOCAL0
  LogLevel INFO
• Save and Close the file.
• Restart the Service:
  Run the following command in Command Prompt to apply changes:
  Restart-Service sshd
• Collect Logs (After the next backup fails): The logs will now appear in the following location:
  C:\ProgramData\ssh\logs\sshd.log