"Inventory backup failed" observed in NSX Manager logs

Products

• VMware NSX

Issue/Introduction

• Windows SFTP file server is used for NSX Backups.
• OpenSSH version on Windows Server is 9.5.
• NSX Manager backup will complete successfully.
• Manager logs may indicate a silent failure:
  /var/log/proton/nsxapi.log:
2024-11-10T08:00:57.373Z <nsx_manager_hostname> NSX 67474 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Backup file to copy: /image/backup/node/1731225646.tar
2024-11-10T08:00:57.374Z <nsx_manager_hostname> NSX 67474 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Copying node backup file, location: sftp://192.168.111.222/<backup_directory>/cluster-node-backups/3.2.3.1.0.22104638-########-#########-####-########2feb-<manager_ip_address>/backup-2024-11-10T08_00_00UTC/node_backup-########-#########-####-########2feb-<manager_ip_address>.tar
..
2024-11-10T08:01:07.383Z <nsx_manager_hostname> NSX 70181 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP29256" level="ERROR" subcomp="manager"] 
2024-11-10T08:01:07.383Z <nsx_manager_hostname> NSX 70181 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP29111" level="ERROR" subcomp="manager"] Error while generating inventory summary
2024-11-10T08:01:07.416Z <nsx_manager_hostname> NSX 70181 - [nsx@6876 comp="nsx-manager" errorCode="MP29111" level="ERROR" s2comp="backup-restore" subcomp="manager"] Inventory backup failed with ; BackupAsyncStatus [BackupStatus [status=URI_NOT_FOUND, statusDetail=Specified uri '/<backup_directory>/inventory-summary/3.2.3.1.0.22104638-########-#########-####-########2feb-<manager_ip_address>/inventory-2024-11-10T08_01_01UTC.json' not found., remoteUri=sftp://192.168.111.222:22/<backup_directory>/inventory-summary/3.2.3.1.0.22104638-########-#########-####-########2feb-<manager_ip_address>/inventory-2024-11-10T08_01_01UTC.json, errorCode=null, startTime=1731225665255, endTime=1731225667383]; responseBody=null].
..
2024-11-10T08:05:00.684Z <nsx_manager_hostname> NSX 70181 - [nsx@6876 comp="nsx-manager" level="INFO" s2comp="backup-restore" subcomp="manager"] Inventory backup completed successfully.

Environment

• VMware NSX-T Data Center 3.2.x
• VMware NSX 4.x

Cause

• When NSX Manager tries to copy the backup file from Linux to Windows, it is also trying to preserve file properties.
• OpenSSH in Windows won't allow the Manager to copy the file and retain it's properties, which will cause this failure. 

Resolution

This is a known issue impacting NSX Manager backup when Windows Server is used as the file server.

To workaround the issue, the following needs to be changed on all NSX Managers:

1. SSH to all NSX Managers as root.
2. Back up the python file:
   cp /opt/vmware/nsx-node-api/bin/python/management_api/napi/root/node/file_store/__self__.py /tmp/__self__.py
3. Edit the file and find the following line:
   cmd_list, sftp_put_cmd = ssh_copy_command_list(
    protocol, port, usr, path, (server, req_obj["uri"]))
4. Delete these two lines.
5. In place of deleted lines, insert the following (please keep the indentation):
   if '/var/vmware/nsx/file-store/inventory' in path:
    cmd_list, sftp_put_cmd = ssh_copy_command_list(
        protocol, port, usr, path, (server, req_obj["uri"]), None, False)
else:
    cmd_list, sftp_put_cmd = ssh_copy_command_list(
        protocol, port, usr, path, (server, req_obj["uri"]))
6. Restart mp-api server:
   service nsx-mp-api-server restart
7. Restart proton service:
   /etc/init.d/proton restart

Please note this workaround is specific for NSX-T Data Center 3.2.3.x.
If you encountered this issue with other version of NSX, please open a new case with reference to this KB article to confirm a workaround in your product version.