Multiple backups are being generated in NSX with 'Detect NSX configuration change' feature enabled
search cancel

Multiple backups are being generated in NSX with 'Detect NSX configuration change' feature enabled

book

Article ID: 432210

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • Multiple cluster backups are being generated by manager node despite no recent configuration change.

  • 'Detect NSX configuration change' feature is enabled for backups.

  • Manager logs are analyzed as below:

    • Manager issues desired-config command to get the last modified time from all nodes
      2026-02-19T12:05:07.312Z  INFO scheduling-worker-1 NapiBackupGenerationServiceImpl 2349253 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Issuing command http://localhost:7441/api/v1/cluster/########-####-####-####-############/cluster/desired-config
2026-02-19T12:05:07.313Z  INFO ForkJoinPool.commonPool-worker-29061 NapiBackupGenerationServiceImpl 2349253 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Issuing command http://localhost:7441/api/v1/cluster/########-####-####-####-############/cluster/desired-config
2026-02-19T12:05:07.313Z  INFO ForkJoinPool.commonPool-worker-29062 NapiBackupGenerationServiceImpl 2349253 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Issuing command http://localhost:7441/api/v1/cluster/########-####-####-####-############/cluster/desired-config


    • Logs indicate a mismatch in lastModifiedTime across manager nodes, triggering ClusterNodeOnChangeBackupJob.
      2026-02-19T06:05:08.109Z  INFO ForkJoinPool.commonPool-worker-28951 DesiredStateChangeBackupHandler 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] lastModifiedTime 2026-02-19T06:05:07.888
2026-02-19T07:05:08.091Z  INFO ForkJoinPool.commonPool-worker-28970 DesiredStateChangeBackupHandler 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] lastModifiedTime 2026-02-19T07:05:07.731
2026-02-19T08:05:08.023Z  INFO ForkJoinPool.commonPool-worker-28989 DesiredStateChangeBackupHandler 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] lastModifiedTime 2026-02-19T08:05:07.754
2026-02-19T09:05:08.466Z  INFO ForkJoinPool.commonPool-worker-29008 DesiredStateChangeBackupHandler 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] lastModifiedTime 2026-02-19T09:05:08.385
2026-02-19T11:05:07.790Z  INFO ForkJoinPool.commonPool-worker-29043 DesiredStateChangeBackupHandler 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] lastModifiedTime 2026-02-19T11:05:07.623
2026-02-19T12:05:07.805Z  INFO ForkJoinPool.commonPool-worker-29062 DesiredStateChangeBackupHandler 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] lastModifiedTime 2026-02-19T12:05:07.632

      2026-02-19T12:05:07.310Z  INFO scheduling-worker-1 ClusterNodeOnChangeBackupJob 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Scheduler starting to create and upload cluster backup
2026-02-19T12:05:07.835Z  INFO scheduling-worker-1 ClusterNodeOnChangeBackupJob 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Create and upload cluster backup
2026-02-19T12:05:45.264Z  INFO scheduling-worker-1 ClusterNodeOnChangeBackupJob 2349253 - [nsx@6876 comp="nsx-manager" level="INFO" s2comp="backup-restore" subcomp="manager"] Cluster backup completed successfully.

       

    • Audit logs of one or more manager nodes (/var/log/nsx-audit-write.log) show frequent POST, PATCH, or DELETE operations
      /var/log/nsx-audit-write.log
2026-02-19T12:05:06.579Z mgr03 NSX 628491 - [nsx@6876 audit="true" comp="nsx-manager" depr="true" level="INFO" method="POST" path="/api/v1/b
atch" subcomp="manager" update="true"] UserName="username", Src="##.##.##.##", ModuleName="Batch", Operation="RegisterBatchRequest", 
Operation status="success", New value=[{"atomic":false} {"continue_on_error":true}]
2026-02-19T12:05:07.301Z mgr03 NSX 628491 - [nsx@6876 audit="true" comp="nsx-manager" depr="true" level="INFO" method="POST" path="/api/v1/b
atch" subcomp="manager" update="true"] UserName="username", Src="##.##.##.##", ModuleName="Batch", Operation="RegisterBatchRequest", 
Operation status="success", New value=[{"atomic":false} {"continue_on_error":true}]
2026-02-19T12:05:07.632Z mgr03 NSX 628491 - [nsx@6876 audit="true" comp="nsx-manager" depr="true" level="INFO" method="POST" path="/api/v1/b
atch" subcomp="manager" update="true"] UserName="usernamel", Src="##.##.##.##", ModuleName="Batch", Operation="RegisterBatchRequest", 
Operation status="success", New value=[{"atomic":false} {"continue_on_error":true}]

Environment

VMware NSX

Cause

When the Detect NSX configuration change option is enabled, the NSX Manager monitors the "Last Modified" timestamp of the cluster configuration. Any API operation that modifies the system state—including automated tasks, script-based batch operations, or runtime state updates from external integrations will update this timestamp.

The NSX Manager scheduler periodically compares the local lastModifiedTime with the cluster's desired-config. If any node reports a newer timestamp due to these operations, a full configuration backup is triggered to ensure the backup reflects the most recent state.

Resolution

To resolve or mitigate the frequency of these backups, perform the following:

  1. Identify the Source of Changes:

    • Examine /var/log/nsx-audit-write.log on the NSX Manager nodes.

    • Search for update="true" and identify the Src="##.##.##.##" (Source IP) and UserName associated with the frequent POST/PATCH/DELETE calls.

  2. Evaluate API Operations:

    • Determine if the identified API calls are critical. Frequent "Read-Write" operations that do not change actual functional configuration but update the metadata/timestamp will still trigger the backup.

    • If the source is an automated integration, consider increasing the polling interval of that tool.

  3. Adjust Backup Strategy:

    • If the environment has constant, expected state changes, the "Detect NSX configuration change" may be too sensitive.

    • Workaround: Disable "Detect NSX configuration change" and instead configure a Weekly or Daily scheduled backup at a specific time to reduce the load on the storage and manager CPU.

Additional Information

Reference documents:

Start or Schedule Backups

Powered by Wolken Software