VCF deployment fails on "Deploy and configure NSX" when the "Configure NSX Backup Schedule" task fail

Products

• VMware SDDC Manager
• VMware NSX

Issue/Introduction

• In the VCF Installer UI, the "Deploy and configure NSX" step shows a failure
• In the Task details pane, the "Configure NSX Backup Schedule" has a status of Failed
  • The details of this failed task are similar to the following:
    
    Failed to configure NSX Backup Schedule for NSX Managers - [########] Reference Token: ######
    
    
• On the SDDC Manager appliance, the backup user has multiple failed logins recorded:
  
  root@v###### [ ~ ]# faillock
Login           Failures    Latest failure         From
backup              3       2025-05-21 17:11:25    ##.##.##.##
postgres            0
root                0
vcf                 0       
vcf_commonsvcs      0
vcf_domainmanag     0
vcf_sos             0

Note: After three consecutive failed logins, the backup user account is locked.
  
  
• Messages similar to the following are seen in the domainmanager.log file on the VCF Installer VM:
  
  2025-05-07T11:24:26.179+0000 ERROR [vcf_dm,#############################,5dbc] [c.v.v.c.n.s.c.c.ComplexHelpers,dm-exec-27]  Exception occurred during NSX API invocationjava.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) (statusCode:400) => {    messages = [],    data =  => {error_message=Authentication failed on fileserver sftp://<SDDC Manager FQDN>:22/nfs/vmware/vcf/nfs-mount/backup., httpStatus=BAD_REQUEST, error_code=29115, module_name=backup-restore},    errorType = INVALID_REQUEST}

  
  
• On the SDDC Manager VM, using the lookup_passwords command shows that the saved password for the backup user is configured as expected
  
  

  vcf@###### [ ~ ]$ lookup_passwords

  Password lookup operation requires ADMIN user credentials. Please refer VMware Cloud Foundation Administration Guide for setting up ADMIN user.

  Supported entity types: ESXI VCENTER PSC NSX_MANAGER NSX_CONTROLLER NSXT_MANAGER NSX_ALB NSXT_EDGE BACKUP VXRAIL_MANAGER AD
Enter an entity type from above list: BACKUP
Enter page number (optional):
Enter page size (optional, default=50):
Enter Username: admin@local
Enter Password:
        BACKUP
        identifiers: <SDDC Manager FQDN>
        workload: site-a
                username: backup
                password: <configured password>
                type: FTP
                account type: SYSTEM

    Page : 1/1, displaying 1 of total 1 entities in a page.
  
  

• Attempting to login as the backup user with the configured password on the SDDC Manager VM, or via SFTP as the backup user to the SDDC Manager VM, fails with an authentication failure.
• There is no backup configured in the NSX Manager deployed via the VCF Installer.
• Messages similar to the following are seen in the syslog file on the NSX Manager:
  
  2025-05-07T11:24:26.168Z ######## NSX 87045 - [nsx@4413 comp="nsx-manager" errorCode="MP29115" level="ERROR" reqId="005d7233-####-####-####-9249038a2180" subcomp="manager" username="admin"] ; BackupAsyncStatus [BackupStatus [status=AUTH_FAILURE, statusDetail=, remoteUri=sftp://<SDDC Manager FQDN>:22/nfs/vmware/vcf/nfs-mount/backup, errorCode=null, startTime=1746617057163, endTime=1746617066162]; responseBody=null]
2025-05-07T11:24:26.173Z ######## NSX 87045 SYSTEM [nsx@4413 audit="true" comp="nsx-manager" level="INFO" reqId="005d7233-####-####-####-9249038a2180" subcomp="manager" update="true"] UserName="admin", Src="##.##.##.##", ModuleName="BackupConfiguration", Operation="ConfigureBackupConfig", Operation status="failure", New value=[{"site_id":"localhost","frame_type":"LOCAL_LOCAL_MANAGER"} {"backup_enabled":true,"backup_schedule":{"seconds_between_backups":3600,"resource_type":"IntervalBackupSchedule"},"inventory_summary_interval":240,"remote_file_server":{"server":"<SDDC Manager FQDN>","port":22,"protocol":{"protocol_name":"sftp","ssh_fingerprint":"SHA256:Rm3TsKpRTDWMeVCDgUmB5XD################","authentication_scheme":{"scheme_name":"PASSWORD","username":"backup"}},"directory_path":"/nfs/vmware/vcf/nfs-mount/backup"}}]
2025-05-07T11:24:26.175Z ######## NSX 87045 SYSTEM [nsx@4413 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName:'admin' ModuleName:'backup-restore' Operation:'PUT@/api/v1/cluster/backups/config' Operation status: 'failure' Error: Authentication failed on fileserver sftp://<SDDC Manager FQDN>:22/nfs/vmware/vcf/nfs-mount/backup.

Environment

• VCF 9.0
• VMware NSX 9.0
• VCF 5.x 

Cause

The backup user password configured and saved by the VCF Installer is not set properly in the OS of SDDC Manager VM. When the backup is configured in NSX, a test SFTP connection is made from the NSX Manager to the SDDC Manager as the backup user using the configured password. This operation is attempted multiple times and results in the task failing and the backup user account being locked.

Resolution

This is a known issue affecting VCF version 5.x-9.x and VMware NSX 4.x-9.x and there is currently no resolution.

Workaround:

Log in to the SDDC Manager VM CLI as the vcf user and elevate to root user via the su - command.

Issue the following command to unlock the backup user account:

faillock --user backup --reset 

Lookup the backup user's password by issuing the following command:  

lookup_passwords

Reset the password by issuing the command:  

passwd backup 

Confirm the password you entered.

Restart the VCF deployment.