"Authentication failed", "bad directory" or "server disk full" error when preforming NSX backup to Windows SFTP server
search cancel

"Authentication failed", "bad directory" or "server disk full" error when preforming NSX backup to Windows SFTP server

book

Article ID: 312385

calendar_today

Updated On:

Products

VMware NSX VMware NSX-T Data Center

Issue/Introduction

  • NSX backups fail with errors such as "bad directory path" or "sftp server disk full" or "authentication failed to fileserver":

  • OR we may see the "Either backup server login failed or unauthorized access to backup directory":

 

  • OR we may see the  "authentication failed to fileserver" error as shown below:

  • There is available disk space on the SFTP server.
  • The backup directories (i.e. cluster-node-backups etc.) are created correctly on the SFTP server.
  • The SFTP server used is running on a Window server.
  • SFTP server is reachable and SSH and SFTP connections to server do not have any issue.
  • There is no permission issue found on backup directory, Read Write access is set on parent directory, as per the NSX Administration Guide.
  • In the NSX Manager log /var/log/syslog you see the following errors:


####-#### NSX 5266 - [nsx@6876 comp="nsx-manager" errorCode="MP29111" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with Status [status=URI_NOT_FOUND, statusDetail={"module_name": "node-services", "error_message": "Specified uri '/Users/Administrator/nsxbackup/#####/cluster-node-backups/4.1.2.#####-74af0842-####-####-####-adf3811321b5-192.168.#.10/backup-YYYY-MM-DDT10_59_07UTC/cluster_backup-74af0842-####-####-####-adf3811321b5-192.168.#.#-nsx-policy-manager.tar' not found.", "error_code": 36219}, startTime=15########704, endTime=157#######560].

####-#### NSX 86061 - [nsx@6876 comp="nsx-manager" errorCode="MP29115" level="ERROR" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with ; BackupAsyncStatus [BackupStatus [status=AUTH_FAILURE, statusDetail=Cannot connect to sftp server. The remote server returned error: status: 255   out: '[Errno 5] Input/output error'.

Environment

  • VMware NSX-T Data Center
  • VMware NSX

Cause

  • The issue occurs due to Windows servers maximum path length limit of 260 characters, the filename can only be 256 characters, as 4 are reserved for drive and NULL terminating character, see Maximum Path Length Limitation for more details.

Note: It has also been seen, when using Windows OpenSSH versions lower than the supported 8.1 version, there is also a 260 character limitation.

  • To identify the length of the path used by NSX, for a backup on the SFTP, review the log /var/log/nvpapi/api_server.log on the NSX manager which was VIP leader when the backup was attempted. You will see a log line similar to:

PUT /image/backup/cluster/backup_<UUID>_nsx-manager.tar /home/backup/cluster-node-backups/<NSX-version-UUID-IP/FQDN>/backup-<date-time>/cluster_backup-<UUIDIP/FQDN>-nsx-ufo-backup-restore.tar

Resolution

  • Ensure you use the supported versions of Windows and OpenSSH. as per documentation:

VMware NSX 4.2 Configure Backups
VMware NSX-T Datacenter 3.2 Configure Backups

  • Enable long paths in Windows server:
    • Edit the local group policy on the SFTP server and enable win32 long path in local group policy to remove the limit and also reboot the server for this to take effect OR Contact Windows support to extend the Windows maximum path length limit of 260 characters, refer to Windows documentation for steps.
    • Microsoft Windows File Management - Maximum Path Length Limitation
  • The Windows Maximum Path Length configuration change will only be effective if running OpenSSH 8.1 or higher, lower versions of OpenSSH will continue to be restricted to 260 characters even after the Windows Maximum Path Length configuration change.

 

Test SFTP backup server:

A method to test if the SFTP server connection is working, correct permissions are applied and no file path length restrictions apply, is to use the following steps:

  1. Log into the NSX manager as root user and cd to /root/. 
  2. Create a tmp file: touch tmp.txt
  3. Connect through SFTP to the Windows sftp server :
    • sftp <backup_user>@<sftp_server_ip>
  4. Then execute these commands:
    • mkdir <backup_dir_given_in_backup_config>
    • mkdir <backup_dir_given_in_backup_config>/cluster-node-backups
    • mkdir <backup_dir_given_in_backup_config>/cluster-node-backups/#.#.#.0.21061376-Dual-########-####-####-####-############-###.###.###.###-########-####-####-####-############
    • mkdir <backup_dir_given_in_backup_config>/cluster-node-backups/#.#.#.0.21061376-Dual-########-####-####-####-############-###.###.###.###-########-####-####-####-############/backup-####-##-##T##_##_##UTC
    • PUT tmp.txt <backup_dir_given_in_backup_config>/cluster-node-backups/#.#.#.0.21061376-Dual-########-####-####-####-############-###.###.###.###-########-####-####-####-############/backup-####-##-##T##_##_##UTC/cluster_backup-########-####-####-####-############-###.###.###.###-########-####-####-####-############-nsx-ufo-backup-restore.tar

Note: <backup_dir_given_in_backup_config> is the backup directory path specified in the NSX UI.
It is also expected to get a failure result, is the directory already exist, NSX ignores this error.

The details of the file path are not that important, the above example has the IP address, UUID and date/time hashed out, but can be used as is.

Additional Information

  • After an NSX upgrade to NSX 4.x, previous working backups, before the 4.x upgrade and using a Windows backup server, may fail.  In NSX 4.x versions, the backup file has more detail, which means the backup file which is stored on the backup server will be longer I.E. more characters. This results in the file path now being longer.
  • Ensure the destination server supports the full length of the backup file, in the folder of the destination backup server.
  • For additional information see Troubleshooting NSX Backup and Restore Failures
Powered by Wolken Software