NSX Backups to Windows SFTP server fails with error "Cannot connect to sftp server"

Products

• VMware NSX

Issue/Introduction

• NSX manual and scheduled backups fail intermittently
• The issue is seen only when using the Windows SFTP server for NSX backups
• sshd service is confirmed to be running on the Windows server by using the command
  Get-Service -Name ssh-agent, sshd
• NSX manager can login into the SFTP server via CLI
• The SFTP version is compatible with the NSX manager
  
  
• During a failure event, NSX manager /var/log/nvpapi/api_server.log reported error similar to the below
  
  napi.root.node.file-store.utils ERROR Unexpected ssh output: Connection timed out during banner exchange
napi.root.node.file-store.utils ERROR Unexpected ssh output: Connection closed
Connection closed.
napi.root.node.file-store.utils ERROR OSError processing ssh command [Errno 5] Input/output error
Traceback (most recent call last):
  File "/opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/file_store/utils.py", line 617, in _ssh_cmd
    output = os_read(fd, 1024, True)
  File "/opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/file_store/utils.py", line 582, in os_read
    output = do_os_read(fd, n)
  File "/usr/lib/python3/dist-packages/gevent/os.py", line 100, in nb_read
    result = _read(fd, n)
OSError: [Errno 5] Input/output error
  
  
• NSX manager /var/log/proton/nsxapi.log reported error similar to the below
  
  2024-10-25T08:20:30.603Z ERROR http-nio-127.0.0.1-7440-exec-627 LocalBackupConfigurationFacadeImpl 86869 - [nsx@6876 comp="nsx-manager" errorCode="MP29115" level="ERROR" reqId="2a#####d-####-####-####-5#####5b3526" s2comp="backup-restore" subcomp="manager" username="admin"] Cluster backup failed with ; BackupAsyncStatus [BackupStatus [status=AUTH_FAILURE, statusDetail=Cannot connect to sftp server. The remote server returned error: status: 255   out: '[Errno 5] Input/output error'., remoteUri=sftp://#.#.#.#:22/NSXBackup/backups, errorCode=null, startTime=1729844427083, endTime=1729844430583]; responseBody=null].

2024-10-25T08:20:30.603Z ERROR http-nio-127.0.0.1-7440-exec-627 LocalBackupConfigurationFacadeImpl 86869 - [nsx@6876 comp="nsx-manager" errorCode="MP29115" level="ERROR" reqId="2a41####-437b-####-8072-####b85b3526" subcomp="manager" username="admin"] ; BackupAsyncStatus [BackupStatus [status=AUTH_FAILURE, statusDetail=Cannot connect to sftp server. The remote server returned error: status: 255   out: '[Errno 5] Input/output error'., remoteUri=sftp://#.#.#.#:22/NSXBackup/backups, errorCode=null, startTime=1729844427083, endTime=1729844430583]; responseBody=null]
  
  
• C:\ProgramData\ssh\logs\sshd.log in the Windows sftp server reported error similar to the below
  
  Connection from #.#.#.# port 57968 on #.#.#.# port 22
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.11
debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.11 pat OpenSSH* compat 0x04000000
debug2: fd 4 setting O_NONBLOCK
debug3: spawning "D:\\Apps\\OpenSSH-Win64\\sshd.exe" -y as user
debug2: Network child is on pid 21252
debug3: send_rexec_state: entering fd = 6 config len 2316
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug3: ssh_msg_send: type 0
debug3: ssh_msg_send: type 0
debug3: preauth child monitor started
debug3: append_hostkey_type: ssh-rsa key not permitted by HostkeyAlgorithms [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: send - WSASend() ERROR:10053, io:000001201C6A0EF0 [preauth]
ssh_dispatch_run_fatal: Connection from #.#.#.# port 57968: Connection aborted [preauth]

Environment

VMware NSX 4.1.2.5

Cause

This is an issue with the Windows SFTP server 

Resolution

This is not an issue with VMware NSX.

To resolve the issue, fix the issue with the Windows SFTP server. 

Reference: SFTP connectivty issue reported with regard to KB5044277 - Cumulative update for Oct'24 for Win2k19 std

Workaround:

Configure backups to a Linux SFTP server temporarily to make sure we have latest NSX backups available. 

Additional Information

Backing Up NSX Manager: Configure Backups