Loss of synchronization between NSX Manager nodes

Products

• VMware NSX

Issue/Introduction

Policy changes, such as changes to firewall rules and groups made in the NSX Global Manager do not take effect at the Local Manager level.

Inconsistent data is observed within the local NSX Manager cluster.

This behavior can be observed even when the NSX Manager user interface reports that the synchronization status is OK.

Cause

Certain conditions may cause synchronization loss between NSX Manager nodes.

Resolution

Remediation can be attempted using these steps: 

• Performing the command restart service async_replicator from the NSXCLI (admin login) of each manager
• Performing the commands start search resync all and start search resync telemetry from NSXCLI
• Selecting "Force Sync" from the Site Manager page on the Global Manager interface

Refer to knowledge base article Resynchronization commands for NSX-T Local and Global Managers for more information.

However, in certain cases, these steps do not address the issue. In this scenario, a restart can resolve the problem.

If policy state inconsistency remains:

• Perform a rolling restart of all local NSX Manager nodes.
  To maximize service availability, reboot one manager at a time and wait for it to start fully before proceeding to the next one
• If Federation is configured, perform a rolling restart of all Global Manager nodes

The restart can be initiated with the command reboot on the NSXCLI or by using the soft restart option in vSphere.