NSX-T scheduled backups stop working

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX-T 2.5.x and 3.0.x environments.
• NSX-T scheduled backups initially work but stop working after a number of days.
• At the time of the expected scheduled backup, the FTP server SSH logs (secure-*) display message(s) similar to:

Aug 4 02:32:12 pmav0109 sshd[26179]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=sftp-server.com user=svcvmware

 

• NSX Manager logs (syslog) display message(s) similar to:

2020-08-04T00:31:42.293Z napi.root.node.file-store.utils ERROR Unexpected ssh output: The authenticity of host 'sftp-server.com (10.10.10.10)' can't be established.
2020-08-04T00:31:42.297Z napi.root.node.file-store.utils ERROR Unexpected ssh output: Are you sure you want to continue connecting (yes/no)? Warning: Permanently added 'sftp-server.com,10.10.10.10)' (ECDSA) to the list of known hosts.

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

In the event that the SFTP backup server rejects the NSX Manager connection following an initial successful connection, the NSX Manager client may be unable to reconnect. Backups then stop working.

Resolution

This issue has been resolved in NSX-T Data Center 2.5.2.2 and 3.1.0.0.

Workaround:
As a temporary workaround, restart all 3 NSX Manager VMs.
The scheduled backup will resume however the issue may re-occur after a period of time.
Therefore an upgrade to the fixed version is recommended.