Intermittently NSX backup failures for scheduled, manual and Upgrade PRECHECKs to SFTP Server

Products

• VMware NSX

Issue/Introduction

• You have multiple NSX-T deployment scheduled for daily backups
• Multiple NSX-T backup scheduled to run daily results in intermittent authentication failures.
• In the /var/log/proton/nsxapi.log you will see the following error:

2025-08-20T18:34:53.907Z  INFO http-nio-127.0.0.1-7440-exec-24 NapiBackupGenerationServiceImpl 5510 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="########- ####-####-####-df562fe1073d" subcomp="manager" username="nsx-opsagent"] Exception received from napi statusDetail
 {
  "error_code": 36222,
  "error_message": "The remote server returned error: status: 255   out: '[Errno 5] Input/output error'.",
  "module_name": "node-services"
}
2025-08-20T18:34:54.024Z ERROR http-nio-127.0.0.1-7440-exec-24 NapiBackupGenerationServiceImpl 5510 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP29256" level="ERROR" reqId="########- ####-####-####-df562fe1073d" subcomp="manager" username="nsx-opsagent"]
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "{<EOL>  "error_code": 36222,<EOL>  "error_message": "The remote server returned error: status: 255   out: '[Errno 5] Input/output error'.",<EOL>  "module_name": "node-services"<EOL>}"
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:101) ~[?:?]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:168) ~[?:?]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[?:?]
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[?:?]
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:825) ~[?:?]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:783) ~[?:?]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717) ~[?:?]
        at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:474) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.callHelper(NapiBackupGenerationServiceImpl.java:446) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.listRemoteFiles(NapiBackupGenerationServiceImpl.java:732) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.BackupRestoreSFTPServiceImpl.getClusterBackupTimestamps(BackupRestoreSFTPServiceImpl.java:218) ~[?:?]
        at com.vmware.nsx.management.backup.service.impl.BackupRestoreSFTPServiceImpl.listClusterBackupTimestamps(BackupRestoreSFTPServiceImpl.java:426) ~[?:?]
        at com.vmware.nsx.management.restore.facade.ClusterRestoreFacadeImpl.listClusterBackupTimestamps_aroundBody2(ClusterRestoreFacadeImpl.java:95) ~[?:?]
        at com.vmware.nsx.management.restore.facade.ClusterRestoreFacadeImpl$AjcClosure3.run(ClusterRestoreFacadeImpl.java:1) ~[?:?]
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[?:?]

Environment

VMware NSX 4.X

Resolution

1. Stagger NSX-T backup so they are not running concurrently in the NSX Backup configuration
2. Remove the Match Group administrators block from sshd_config

• Steps to remove the Match Group administrators block:

• 1. Locate your sshd_config file:

  • On Linux, this is typically /etc/ssh/sshd_config. 
     
  • On Windows, it's usually located at %programdata%\ssh\sshd_config. 
     
• 2. Open the file in a text editor:

  You'll need administrator or root privileges to edit this file.

  • On Linux: sudo nano /etc/ssh/sshd_config (or use your preferred editor like vi or vim). 
     
  • On Windows: Open a PowerShell or Command Prompt as Administrator, then use notepad.exe %programdata%\ssh\sshd_config. 
     
• 3. Find and delete or comment out the block:

  • Look for lines that start with Match Group administrators.
  • To disable the block, you can delete all the lines associated with it. 
     
  • Alternatively, you can comment out the lines by adding a # symbol at the beginning of each line within the block. This keeps the configuration intact but prevents sshd from using it. 

  4. Save this file and exit the text editor

  5.  Restart the sshd service to apply the changes

  on linux (systemd)  :  sudo systemctl restart sshd

  On Windows:  Restart-service sshd

      3.  Uninstall Microsoft Hotfix KB5044277