The vSphere Distributed Switch configuration on some hosts differed from that of VMware vCenter- "Out of Sync"

Products

• VMware vSphere ESXi

Issue/Introduction

• You may see this banner alert when you are viewing the Summary tab of a vSphere Distributed Switch (vDS) -- after selecting the Networking icon view in vCenter (top left of screen):

The vSphere Distributed Switch configuration on some hosts differed from that of the vCenter Server.

• When viewing a list of hosts attached to the distributed switch, or from the manage hosts wizard under the vDS Status column, you see a yellow warning icon and the message:

Out of sync

Environment

vCenter

Cause

BACKGROUND:

• vSphere Distributed Switches (vDSs) are created, and updated, using vCenter. 
• A copy of the vDS is exported to each ESXi host which is attached to the vDS, during a "host sync" operation. 
• It is common for a "host sync" to happen every minute or two, but can sometimes occur more frequently.

 

TYPES OF "OUT OF SYNC" EVENTS:

1) INTERMITTENT / TEMPORARY:

• If network connectivity is interrupted between the vCenter Server and one or more hosts, a synchronization interval may be missed resulting in this alert being displayed.
• This type of interruption can occur during vCenter Service restarts, vCenter Server reboots as well as ESX/ESXi host reboots or network maintenance.
• If it is intermittent / temporary, you should expect the message to disappear within a few moments. 

2) NON-TEMPORARY:

• If the message persists for any longer than a few moments, then the situation will be that the vCenter database tables associated with that specific vDS are logically inconsistent with one another.
• The "Show Details" option associated with the message will give the details of the scope of the inconsistency.
• The "Rectify" option is unlikely to resolve the issue if it is non-temporary. 

 

Resolution

1) INTERMITTENT / TEMPORARY SCENARIO:

• Most of the time, this message is benign and can be safely ignored until a maintenance window.
• Within several minutes, the host's Distributed Switch information should synchronize with vCenter Server, and the warning will clear on its own.

2) NON-TEMPORARY SCENARIO:

• The situation with an "Out of Sync" that is not temporary, is that certain tables in the vCenter database associated with the vDS in question, are logically inconsistent with one another.
• In the normal case (in sync), the vCenter database is meant to be the source of truth for the vDS, and each ESXi host which is attached to the vDS will have a proxy copy of the vDS that agrees with the vCenter records. 
• The ultimate resolution for this "Out of Sync" condition is to get the tables in the vCenter database associated with the vDS in question, to be logically consistent with one another. 
• This means, then, that this is a vCenter database issue, and the ESXi networking on the host becomes the source of truth until the database is corrected.

STEP 1:  SCOPING THE ISSUE

• The first step in scoping the issue is to document exactly what data is mentioned in the "Show Details" output (ESXi host names, ports, etc.)
• The details should be copied from the message "Show Details" output into your notes, and should form part of a Problem Statement for a Broadcom Case, if one is filed.

STEP 2:  ANSWER KEY QUESTIONS

• From the Networking icon view of vCenter, use the Add / Manage Hosts action --> Manage Host Networking and document the following information for each of the impacted hosts
  • The mapping of Physical Adapters to Uplinks for the host (Example:  vmnic0 --> Uplink 1, and vmnic1 --> Uplink 2, etc.)
  • The details for the vmkernel Adapters (Example, IP information, and dvPort Group assignment for each vmkernel adapter)
  • These details will be needed at a later stage of the resolution process.

STEP 3:  FOR ESXi HOSTS THAT ARE NOT PART OF A VSAN AND/OR NSX PREPARED CLUSTER

• If any of the ESXi hosts listed in the "Show Details" output, are NOT part of a VSAN cluster, or an NSX prepared cluster, then the approach below can be attempted for one host at a time.
  • In the Host and Clusters view, place the ESXi host into Maintenance Mode 
  • Right-click on the ESXi host, select Connection --> Disconnect to disconnect the host from vCenter
  • Make sure that you have the appropriate credentials required to add the ESXi host to vCenter inventory.
  • Once the host is in "Disconnected" state from vCenter, remove the host from vCenter's Inventory.
  • The removal from vCenter inventory will remove all entries associated with that ESXi host from the vCenter database.
  • The side effect of this is that it will remove the host from the vDS as well.  
  • However, the ESXi host will still have a copy of the vDS, which can be seen by SSHing into the host and running a command such as "esxcfg-vswitch -l", for example. 
  • The next step is to add the host in the usual way to vCenter, in the Hosts and Clusters icon view.
  • Once the host is added, then switch to the Networking icon view, and use the "Add / Manage Hosts" action to "Add Host" to the vDS.
  • Follow the Add Host Wizard, entering the information that you obtained earlier in this step, for the Physical Adapters, and the vmkernel Adapter dvPort Group assignments.
  • After you click Finish, you should expect to see the host now part of the vDS again, as if the issue had not happened. 
  • Repeat this process for each host named earlier in the Show Details output.

FOR ESXi HOSTS THAT ARE PART OF A VSAN AND/OR NSX PREPARED CLUSTER:

• Open a case with Broadcom Support following the instructions documented in the KB Creating and managing Broadcom support cases
• In your Problem Statement, please provide the following information:
  • FQDN and Management vmkernel IP address of each ESXi host in the Show Details output
  • Any other details from the Show Details output
  • If the host is part of a VSAN cluster, the cluster name and the total number of ESXi hosts in the cluster
  • If the host is part of an NSX prepared cluster, the cluster name and the total number of ESXi hosts in the cluster
• For Product, choose VMware vCenter Server
• Choose Networking for the Component Name