"The vSphere Distributed Switch configuration on some hosts differed from that of the vCenter Server" seen in the vCenter Networking icon view, Summary tab of a vDS

Products

• VMware vCenter Server

Issue/Introduction

When you select the Summary tab in vCenter when focused on a vDS in the Networking view, "The vSphere Distributed Switch configuration on some hosts differed from that of VMware vCenter" is often referred to as an "Out of Sync" condition.

The "Out of Sync" condition means that one or more database tables associated with the affected vDS within the vCenter database are inconsistent with one another, regarding one or more fields and/or rows in one or more of the tables.

This is often referred to as a "vDS" "Out of Sync" condition.

Symptoms:

• In the vSphere Client, the Summary tab for a vSphere Distributed Switch (vDS) shows: The vSphere Distributed Switch configuration on some hosts differed from that of the vCenter Server.
• It sometimes affects only a single ESXi host, but can affect multiple ESXi hosts as well.
• The Rectify option is greyed out or fails to clear the alert.
• Discrepancies may list specific vNetwork Distributed Ports (e.g., ports 18421, 5508).
• Warning can appear after a vDS upgrade (e.g., from version 6.6 to 7.0.3).
• DVport reconfigure task may fail with the following error:
  "Cannot complete a vSphere Distributed switch operation for one or more host members"

Cause

• The list of causes is not fully understood at this time, although it has been investigated in releases from 6.x onwards to present day. One possibility that has been considered is a database "race condition" which happens sometimes and not others for similar operations. Race conditions are inherently difficult to reproduce and this inhibits root cause determination.
• In addition, the root cause determination is made more difficult in that symptoms often do not show up until well after whatever causes the inconsistency.
• This condition can occur when the vCenter Server database (VCDB) and the ESXi host's proxy copy of the vDS configuration become inconsistent. This can be caused by:
  1. Environmental Issues: A virtual machine NIC holding a stale reference to a dvPort.
  2. Database Race Conditions: Inconsistencies within vCenter database tables.
• Potential causes for DVS "out of sync":
  • Changing the configured MTU on a vDS while LAG is configured can cause LACP FLAPS.
  • VDS info becomes out of sync on hosts if: Envoy Session Limits Exceeded by Replication Services causing ESXi Hosts to show as 'Not Responding'

Resolution

The solution is to get the database tables back in a logically consistent state within the vCenter database.

Workaround:

Phase 1: Environmental Remediation (Non-Disruptive)

Before performing database scripts, verify if the issue is caused by a stale port reference:

1. Navigate to vCenter > Networking > vDS > Ports.
2. Locate the port ID(s) listed in the error details.
3. Check the Connectee column. If a VM is listed, remove the NIC from that VM and add a new NIC associated with the port.
4. If no VM is listed, go to the Statistics tab for the affected Port Group and toggle Auto-refresh to force a synchronization task.

Phase 2: Scripted Database Repair

If Phase 1 does not resolve the alert, follow the scripted remediation provided below.

Prerequisites and Warnings:

1. This should not be used without a Broadcom Networking Support Engineer first evaluating any given situation.
2. This approach will NOT impact connectivity of VMs or Hosts.
3. It is important to follow each step in the sequence indicated.
4. Do not do any additional steps in the exercise.
5. Do not skip any steps.
6. If at any time you observe a result other than expected, STOP and do not attempt resolution. Open a P1 Networking case with Broadcom using the instructions at Creating and managing Broadcom support cases.

STEPS:

1. Declare a short maintenance window for the affected vCenter server.
   NOTE: If the affected vCenter is part of a linked mode configuration, the maintenance window will apply to all vCenters that are linked to the vCenter where the issue is observed.
   • An hour is usually more than sufficient, and it will not affect running VMs in the environment.
   • The maintenance window should be chosen during a time when you are not expecting other applications to do API calls to the vCenter server, such as backups, automation deployments, or any other planned power or migration operations for VMs.
   • This means that unlike many such procedures, normal business hours are good candidates for the window.
2. In preparation for the maintenance window, use WinSCP or a suitable other tool to copy the dvportOutOfSyncSupportV4.tar file (attached to this KB) into the /root directory of the vCenter server. If the vCenter server is part of a linked mode configuration, this need only be done on the vCenter where the issue is observed.
   • After the file is moved, SSH into the vCenter server with the root account, and change directory into /root.
   • Then untar the dvportOutOfSyncSupportV4.tar file by the command:

     tar xf dvportOutOfSyncSupportV4.tar

3. At the start of the maintenance window:
   1. Identify the ESXi host where the vCenter server is running.
   2. Use the vSphere client to log into that ESXi host using the root account.
   3. Power off the vCenter server.
   4. When the vCenter server is fully powered off, take a powered off VM snapshot. This provides a revert point which can then be reverted to, if there are any problems with the procedure. The revert point captures not only the state of vCenter prior to the procedure, but also a functional backup of any vDS' that are part of the vCenter.
      NOTE: If the vCenter is part of a linked mode configuration, do step d) for each of the linked vCenters at this time.
4. After revert point snapshot(s) have been taken, power on the vCenter (and if it is part of a linked mode configuration, the other vCenters also).
5. After all of the services have started normally on the vCenter (and if it is part of a linked mode configuration, the other vCenters also), SSH into the vCenter with the root account.
   NOTE: If the vCenter server is part of a linked mode configuration, this and subsequent steps need only be done on the vCenter where the issue is observed.
6. In the SSH session, make sure that you are in the Bash shell (i.e. enter shell at the initial command prompt if needed).
7. Stop the vmware-vpxd and vmware-content-library service using the following commands:

   service-control --stop vmware-vpxd
   service-control --stop vmware-content-library

8. Change directory to /root, then change ("cd") into the directory dvportOutOfSyncSupport that was created by the "untar" command in Step 2.
9. Run the diagnosis script using the command:

   ./dvportOutOfSyncSupport.sh

   • The script will run very quickly (usually less than a second), and may be so quick that you may think it did not work.
   • There will be a message that indicates "Bad" or "Good" with reference to the logical consistency of the vCenter database tables associated with the vDS.
   • The result of "overallStatus": "Bad" is the expected confirmation of this issue.
   • Take a screen shot of the results for documentation purposes.
10. Change directory back to the directory dvportOutOfSyncSupport that was created by the "untar" command in Step 2.
11. Run the remediation script using the command:

    ./workaround.sh

    Note: As with the diagnosis script, this script will also run very quickly (usually less than a second).
12. Take a screen shot of the results for documentation purposes.
13. Now reboot the vCenter server from the command line using the command:

    reboot

14. The expected result after the reboot, would be that the Out of Sync message is no longer seen when looking at the Summary tab in the Networking view when focused on the vDS. This confirms that the inconsistency on the VC database is resolved.
    • You can check this by logging into the vCenter after it comes up and all the services are started normally. (vCenter > networking icon View > vDS > Summary)
15. The last step is to consolidate (delete) the revert point snapshot(s) we took of the vCenter(s), as the revert point(s) will no longer be needed.
    • This can be done from vCenter itself; no need to log into the host directly as before.
    • If the vCenter is part of a linked mode configuration, consolidate (delete) the revert point snapshot for each of the other vCenters.
    • Do this as soon as possible, since vCenter tends to be a busy VM and the snapshot files would grow considerably large and quickly, which is not a desirable thing for multiple reasons.

Additional Information

IMPORTANT NOTES:

1. It is important to follow each step in the sequence indicated.
2. Do not do any additional steps in the exercise.
3. Do not skip any steps.
4. If at any time you observe a result other than expected, STOP and do not attempt resolution. Open a P1 Networking case with Broadcom using the instructions at Creating and managing Broadcom support cases.