Scheduled backups failing to start after NSX-T Manager restore operation

Products

• VMware NSX

Issue/Introduction

Symptoms:

• Schedule backups will fail on NSX-T Manager but manual backup will work.
• When you check in System —> Backup & Restore tab you will observe that latest schedule backup is failed but when you try to perform manual backup it will work.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

This issue occurs since service which is responsible for taking scheduled backups "QuartzSetupServiceImpl" is down, so scheduled backups will not succeed.

 If automatic backup is already scheduled in customer environment to check if Quartz service is running without any issue you should be able to see below logs in nsxapi.log for every 4 mins.

/var/log/proton# grep "Scheduler starting to create" nsxapi.log 
20XX-XX-XXXXX:XX:XX.XXXX INFO scheduling-worker-1 InventoryBackupJob - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Scheduler starting to create and upload inventory summary 
20XX-XX-XXXXX:XX:XX.XXXX INFO scheduling-worker-1 InventoryBackupJob - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Scheduler starting to create and upload inventory summary

Resolution

This issue is resolved in VMware NSX-T 3.2.2 (build number 20737185)
This issue is resolved in VMware NSX-T 4.0.1.1 (build number 20598726)

Workaround:

1. Find the leader NSX-T manager for QuartzSetupServiceImpl service.

• Below command can be used to find the leader manager for QuartzSetupServiceImpl from any of the Manager note in cluster.

NSXT> get cluster status verbose | find QuartzSetupServiceImpl
    QuartzSetupServiceImpl                     0f2e2842-983a-f74b-77e2-e11156734de6       20163

• As per above output NSX-T manager with UID 0f2e2842-983a-f74b-77e2-e11156734de6 is the leader manager for QuartzSetupServiceImpl.
• To find NSX-T manager with UID 0f2e2842-983a-f74b-77e2-e11156734de6 login to any NSX-T manager with admin account and execute “get cluster status” command and it will list all the manager details with their Name, UID & IP address details. In this case NSXT is the NSX-T manager hostname & 10.172.78.76 is the IP.

NSXT> get cluster status
Cluster Id: c7a0a122-54cc-4078-9297-4f40c45e0902
Overall Status: STABLE

Group Type: DATASTORE
Group Status: STABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    0f2e2842-983a-f74b-77e2-e11156734de6       NSXT                                 10.172.78.76     UP

2. Once you have the leader manager details ssh to that manager with root account and restart Proton service in that manager using below command.

root@NSXT:~# /etc/init.d/proton restart

Additional Information

Impact/Risks:
Impact in this case is scheduled backups will not work and customer might need to take backups manually.