NSX-T backups are failing

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T backups are failing with error "either bad directory path or sftp server disk full" or authentication error.

• There is available disk space on the SFTP server.
• The backup directories (i.e. cluster-node-backups etc.) are created correctly on the SFTP server.
• The SFTP server used is running on a Window server
• SFTP server is reachable and SSH and SFTP connections to server do not have any issue. There is no permission issue found on backup directory
• The SFTP server used is running on a Window server and some Linux distributions using short path. 
• The Windows backup path exceeds 260 characters.
• NSX Manager logs (syslog) display message(s) similar to

Find for Windows; 

#grep -hE "Cluster backup failed with Status.*URI_NOT_FOUND" /var/log/syslog
<179>1 2019-12-14T10:59:38.592Z ####-#### NSX 5266 - [nsx@6876 comp="nsx-manager" errorCode="MP29111" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with Status [status=URI_NOT_FOUND, statusDetail={"module_name": "node-services", "error_message": "Specified uri '/Users/Administrator/nsxbackup/#####/cluster-node-backups/2.4.2.0.0.14269551-74af0842-####-####-####-adf3811321b5-192.168.#.10/backup-2019-12-14T10_59_07UTC/cluster_backup-74af0842-####-####-####-adf3811321b5-192.168.#.10-nsx-policy-manager.tar' not found.", "error_code": 36219}, startTime=1576321162704, endTime=1576321178560].

Following error may also be seen -

2024-03-25T04:44:16.162Z ####-#### NSX 86061 - [nsx@6876 comp="nsx-manager" errorCode="MP29115" level="ERROR" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with ; BackupAsyncStatus [BackupStatus [status=AUTH_FAILURE, statusDetail=Cannot connect to sftp server. The remote server returned error: status: 255   out: '[Errno 5] Input/output error'.

Find for Linux;

<179>1 2020-04-17T08:24:33.684Z ####-#### NSX 18892 - [nsx@6876 comp="nsx-manager" errorCode="MP29111" level="ERROR" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with Status [status=URI_NOT_FOUND, statusDetail={"error_message": "Specified uri '/NSX/cluster-node-backups' not found.", "module_name": "node-services", "error_code": 36219}, startTime=1587111873014, endTime=1587111873677].
Binary file /var/log/syslog matches

Environment

• VMware NSX-T Data Center
• VMware NSX

Cause

• The issue is caused by Windows maximum path length limit of 260 characters.
• There is a 256 character limit for path and filename in Windows.
• The issue is also caused by Linux using short path.

Resolution

There is no resolution as this is not an NSX-T issue. To workaround the issue you can either:

• Extend the Windows maximum path length limit of 260 characters, refer to Windows documentation for steps.
  • Microsoft Windows File Management - Maximum Path Length Limitation
• Contact Linux OS vendor for support.
• Possibly use a different Linux OS distribution that supports a longer filename/path character limit.
• Reduce the size of the SFTP path on the SFTP server.

Additional Information

Note: After upgrade to NSX 4.x previous working backups may fail due to the file path now updated with more characters. This comes from updates in the naming convention from NSX backup.