Scheduled backups failing to start after NSX-T Manager restore operation

Products

• VMware NSX

Issue/Introduction

• Schedule backups fail on NSX-T Manager but manual backup will work.
• When checking System —> Backup & Restore tab, the latest schedule backup is failed but when a manual backup is performed it succeeds.

Environment

VMware NSX-T Data Center

Cause

This issue occurs since service which is responsible for taking scheduled backups "QuartzSetupServiceImpl" is down, so scheduled backups will not succeed.

 If automatic backup is already scheduled in customer environment to check if Quartz service is running without any issue you should be able to see below logs in nsxapi.log for every 4 mins.

/var/log/proton# grep "Scheduler starting to create" nsxapi.log 
20XX-XX-XXXXX:XX:XX.XXXX INFO scheduling-worker-1 InventoryBackupJob - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Scheduler starting to create and upload inventory summary 
20XX-XX-XXXXX:XX:XX.XXXX INFO scheduling-worker-1 InventoryBackupJob - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Scheduler starting to create and upload inventory summary

Resolution

This issue is resolved in VMware NSX-T 3.2.2 (build number 20737185)
This issue is resolved in VMware NSX-T 4.0.1.1 (build number 20598726)

Workaround:

1. Find the leader NSX-T manager for QuartzSetupServiceImpl service.

• Below command can be used to find the leader manager for QuartzSetupServiceImpl from any of the Manager note in cluster.

NSXT> get cluster status verbose | find QuartzSetupServiceImpl
    QuartzSetupServiceImpl                     ########-####-####-############34de6       20163

• As per above output NSX-T manager with UID ########-####-####-############34de6 is the leader manager for QuartzSetupServiceImpl.
• To find NSX-T manager with UID ########-####-####-############34de6 login to any NSX-T manager with admin account and execute “get cluster status” command and it will list all the manager details with their Name, UID & IP address details. In this case NSXT is the NSX-T manager hostname & ###.###.###.76 is the IP.

NSXT> get cluster status
Cluster Id: ########-####-####-####-#######e0902
Overall Status: STABLE

Group Type: DATASTORE
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    ########-####-####-####-#######34de6       NSXT                                 ###.###.###.76           UP

2. Once you have the leader manager details ssh to that manager with root account and restart Proton service in that manager using below command.

root@NSXT:~# /etc/init.d/proton restart

Additional Information

Impact/Risks:
Impact in this case is scheduled backups will not work and customer might need to take backups manually.

For additional information see Troubleshooting NSX Backup and Restore Failures