NSX-T backups are failing with error "either bad directory path or sftp server disk full"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T backups are failing with error "either bad directory path or sftp server disk full"

• There is available disk space on the SFTP server.
• The backup directories (i.e. cluster-node-backups etc.) are created correctly on the SFTP server.
• The SFTP server used is running on a Window server and some Linux distributions using short path. 
  • The Windows backup path exceeds 260 characters.
• NSX Manager logs (syslog) display message(s) similar to

Find for Windows;

#grep -hE "Cluster backup failed with Status.*URI_NOT_FOUND" /var/log/syslog
<179>1 2019-12-14T10:59:38.592Z nsx-mngr-01 NSX 5266 - [nsx@6876 comp="nsx-manager" errorCode="MP29111" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with Status [status=URI_NOT_FOUND, statusDetail={"module_name": "node-services", "error_message": "Specified uri '/Users/Administrator/nsxbackup/testtesttesttes/cluster-node-backups/2.4.2.0.0.14269551-74af0842-d9f9-641f-cc2e-adf3811321b5-192.168.120.10/backup-2019-12-14T10_59_07UTC/cluster_backup-74af0842-d9f9-641f-cc2e-adf3811321b5-192.168.120.10-nsx-policy-manager.tar' not found.", "error_code": 36219}, startTime=1576321162704, endTime=1576321178560].

Find for Linux;

<179>1 2020-04-17T08:24:33.684Z PKS-NSX-MGR03 NSX 18892 - [nsx@6876 comp="nsx-manager" errorCode="MP29111" level="ERROR" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with Status [status=URI_NOT_FOUND, statusDetail={"error_message": "Specified uri '/NSX/cluster-node-backups' not found.", "module_name": "node-services", "error_code": 36219}, startTime=1587111873014, endTime=1587111873677].
Binary file /var/log/syslog matches

Environment

• VMware NSX-T Data Center 2.x
• VMware NSX-T Data Center
• VMware NSX-T
• VMware NSX-T VMware NSX Networking

Cause

• The issue is caused by Windows maximum path length limit of 260 characters.
  • There is a 256 character limit for path and filename in Windows.
• The issue is also caused by Linux using short path.

Resolution

There is no resolution as this is not an NSX-T issue. To workaround the issue you can either:

• Extend the Windows maximum path length limit of 260 characters, refer to Windows documentation for steps.
  • Microsoft Windows File Management - Maximum Path Length Limitation
• Contact Linux OS vendor for support.
  • Possibly use a different Linux OS distribution that supports a longer filename/path character limit.
• Reduce the size of the SFTP path on the SFTP server.

Additional Information

Note: After upgrade to NSX 4.x previous working backups may fail due to the file path now updated with more characters. This comes from updates in the naming convention from NSX backup.