Understanding NSX Manager Backup behaviour: Missing Node Directories in SFTP Repository

Products

• VMware NSX

Issue/Introduction

• In the NSX manager GUI, scheduled backups complete successfully and Last Backup Status is reported as Successful.
• When reviewing the backup repository on the configured SFTP server, backup directories for specific NSX Manager Nodes are missing.

Example: In a three-node cluster consisting of Node 1, Node 2, and Node 3, the SFTP server may only show backup directory for Node 1, while Node 2 and Node 3 lack corresponding directories despite the overall successful cluster backup status.

Environment

VMware NSX

Cause

• NSX manager utilizes Corfu Database which ensures all three NSX Manager nodes remain in a consistent state regarding configuration and data.
• By design, when a scheduled backup is initiated, the backup is only taken from the leader node where the QuartzSetupServiceImpl service is running. As a result, only the leader node's backup directory is visible on the SFTP server for scheduled backups.
• However, if a user logs into an individual manager node and triggers a backup manually, that specific node's backup directory will then become visible in the SFTP repository.

Resolution

• The absence of backup directories for the non-leader nodes during scheduled backups is expected behaviour by design.
• The single leader node's backup is sufficient for restoring the complete three-node NSX Manager cluster in case of a failure.
• If a backup folder is observed for a non-leader node, this indicates that either a manual backup was previously initiated directly from that specific node or that node was previously a leader node.

Additional Information

To know which is the leader node, open a SSH session to one of the manager nodes and login as root user and execute the following commands:

1. su admin -c get cluster status verbose | grep -E "Leaders\:|QuartzSetupServiceImpl"
   1. Copy the UUID of the leader node for service QuartzSetupServiceImpl.
      
2. su admin -c get cluster status | grep <UUID returned in previous command>
   1. This output will show the Leader Node's IP address and hostname.

 

If NSX Manager's Scheduled backup do not execute at the scheduled time but manual backups are successful, refer Knowledge base article 402992.