Upgrade Pre-check failed due to Backup never completes for SDDC/NSX Manager

Products

• VMware NSX

Issue/Introduction

Symptoms

During an SDDC Manager upgrade pre-check, the process fails because an NSX Manager backup task gets stuck. You may observe one or more of the following symptoms:

• SDDC Manager Error: NSX backup check failed. Reason: Read timed out

• SDDC Manager Error: NSX Upgrade coordinator - Failed to suspend backup before upgrade

• NSX Manager UI: The interface displays a continuous "Cluster Backup in Progress" message.

• Behavior: The NSX Manager backup task runs indefinitely without completing or failing.

• Backup Server Verification: No backup files are generated on the backup server during this timeframe, confirming the task is completely stuck.

Environment

VMware Cloud Foundation (VCF) / SDDC Manager

VMware NSX / NSX-T Data Center

Cause

This issue typically occurs due to backend authentication or permission problems halting the backup script, such as:

• The backup user account is locked out on the SDDC Manager.

• There is a password mismatch for the backup user between SDDC Manager and NSX Manager.

• Incorrect ownership or permissions exist on the NFS backup directory.

Resolution

To resolve this issue, you must verify the backup user's status, restart the stuck NSX services, and correct any underlying directory permission issues. Follow these steps sequentially:

Step 1: Check and unlock the backup user account

1. SSH into the SDDC Manager appliance using the vcf user account.

2. Elevate your privileges to root:

   Bash

   su -
   

3. Check if the backup user account is currently locked:

   Bash

   faillock --user backup
   

4. If the account is locked, reset it using the following command:

   Bash

   faillock --user backup --reset
   

Step 2: Verify the backup user password Ensure that the backup user password configured in SDDC Manager exactly matches the one configured within the NSX Manager backup settings. Update the password in either system if a mismatch is found.

Step 3: Stop the stuck NSX backup task

1. SSH into the NSX Manager appliance(s) and log in as root.

2. Restart the API server service to terminate the hung backup process:

   Bash

   systemctl restart nsx-mp-api-server
   

Step 4: Verify and correct NFS backup directory permissions

1. Navigate to the NFS mount path on your backup server or SDDC Manager: /nfs/vmware/vcf/nfs-mount/backup

2. Compare the directory permissions and ownership against a known working site. The backup user must have appropriate read/write access.

3. If permissions or ownership are incorrect, use chmod, chown, and chgrp to correct them.

   Example correction for a directory named cluster-node-backups:

   Bash

   chown backup cluster-node-backups
   chgrp backup cluster-node-backups
   

Once all steps are completed, attempt to run a manual NSX backup to verify the configuration, and then retry the SDDC Manager upgrade pre-check.