NSX-T Backup keeps running indefinitely if the SFTP Server sends a prompt asking for the password

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Backup jobs in the NSX Manager UI appear to be in progress, but never complete.
• NSX-T 3.1.3.1 or recently upgraded to that version.
• The NSX-T Manager logs errors similar to the following in /var/log/nvpapi/api_server.log:

2021-09-13T12:45:03.569Z napi.root.node.file-store.utils ERROR Unexpected ssh output

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

Currently the backup logic checks if the SSH output contains the string "password:" or "Password:" to determine if the remote server is prompting for the password.
This issue occurs if the remote SFTP server being used as the backup destination, returns a string of text after the word "password" followed by the colon in the password prompt when making a SSH connection.
For example the following prompt from a SFTP server would return no match with the search query (password: or Password:), since there are additional strings after the word password, this will be logged as "Unexpected SSH output" message in the logs above:

# Login as: user
# Keyboard-interactive authentication prompts from server: 
# password for user:

Resolution

This issue is resolved in NSX-T Data Center 3.2.0 available at VMware downloads.


Workaround:
Use a different SFTP application server if upgrading is not an option or remove the prompt from the SFTP server authentication check.