"The vSphere Distributed Switch configuration on some hosts differed from that of the vCenter Server" seen in the vCenter Networking icon view, Summary tab of a vDS

Products

• VMware vCenter Server

Issue/Introduction

When you select the Summary tab in vCenter when focused on a vDS in the Networking view, "The vSphere Distributed Switch configuration on some hosts differed from that of VMware vCenter" is often referred to as an "Out of Sync" condition.

The  "Out of Sync" condition means that one or more database tables associated with the affected vDS within the vCenter database are inconsistent with one another, regarding one or more fields and/or rows in one or more of the tables.

Cause

The list of causes is not fully understood at this time, although it has been investigated in releases from 6.x onwards to present day.  One possibility that has been considered is a database "race condition" which happens some times and not others for similar operations.  Race conditions are inherently difficult to reproduce and this inhibits root cause determination.

In addition, the root cause determination is made more difficult in that symptoms often do not show up until well after whatever causes the inconsistency.

 

Resolution

The solution is to get the database tables back in a logically consistent state within the vCenter database. 

 

NOTES:  

1. This should not be used without a Broadcom Networking Support Engineer first evaluating any given situation.  
   
   
2. This approach will NOT impact connectivity of VMs or Hosts

 

STEPS:

1. Declare a short maintenance window for the affected vCenter server.  
   
   NOTE:  If the affected vCenter is part of a linked mode configuration, the maintenance window will apply to all vCenters that are linked to the vCenter where the issue is observed.
   
   
   • An hour is usually more than sufficient, and it will not affect running VMs in the environment.
   • The maintenance window should be chosen during a time when you are not expecting other applications to do API calls to the vCenter server, such as backups, automation deployments, or any other planned power or migration operations for VMs. 
   • This means that unlike many such procedures, normal business hours are good candidates for the window. 
     
     
     
2. In preparation for the maintenance window, use WinSCP or a suitable other tool to copy the "dvportOutOfSyncSupportV4.tar" file (attached to this KB) into the /root directory of the vCenter server.   If the vCenter server is part of a linked mode configuration, this need only be done on the vCenter where the issue is observed.
   
   
   • After the file is moved, SSH into the vCenter server with the root account, and change directory into /root.
   • Then untar the dvportOutOfSyncSupportV4.tar file by the command:
     
     tar xf dvportOutOfSyncSupportV4.tar


3. At the start of the maintenance window:
   
   
   1. Identify the ESXi host where the vCenter server is running.
   2. Use the vSphere client to log into that ESXi host using the root account
   3. Power off the vCenter server
   4. When the vCenter server is fully powered off, take a powered off VM snapshot.  This provides a revert point which can then be reverted to, if there are any problems with the procedure.  The revert point captures not only the state of vCenter prior to the procedure, but also a functional backup of any vDSs that are part of the vCenter. 
      
      NOTE:  If the vCenter is part of a linked mode configuration, do step d) for each of the linked vCenters at this time.  
      
      
      
4. After revert point snapshot(s) have been taken, power on the vCenter (and if it is part of a linked mode configuration, the other vCenters also).
   
   
   
5. After all of the services have started normally on the vCenter (and if it is part of a linked mode configuration, the other vCenters also), SSH into the vCenter with the root account.   
   
   NOTE:  If the vCenter server is part of a linked mode configuration, this and subsequent steps need only be done on the vCenter where the issue is observed.
   
   
   
6. In the SSH session, make sure that you are in the Bash shell (i.e. enter "shell" and the initial command prompt if needed).
   
   
   
7. Stop the vmware-vpxd service using the following command:
   
service-control --stop vmware-vpxd
   
   
   
8. After the vmware-vpxd service is stopped, stop the vmware-content-library service using the following command:
   
   service-control --stop vmware-content-library
   
   
   
9. Change directory to /root, then change ("cd") into the directory "dvportOutOfSyncSupport" that was created by the "untar" command in Step 2.
   
   
   
   
10. Run the diagnosis script using the command:
    
    ./dvportOutOfSyncSupport.sh
    
    
    • The script will run very quickly (usually less than a second), and may be so quick that you may think it did not work.  
    • There will be a message that indicates "Bad" or "Good" with reference to the logical consistency of the vCenter database tables associated with the vDS.
    • The expected result will be "Bad" in this situation.  
    • Take a screen shot of the results for documentation purposes.
      
      
      
11. Change directory back to the directory "dvportOutOfSyncSupport" that was created by the "untar" command in Step 2.
    
    
    
12. Run the remediation script using the command:
    
    ./workaround.sh
    
    
    
13. As with the diagnosis script, this script will also run very quickly (usually less than a second).
    
    
    
14. Take a screen shot of the results for documentation purposes.
    
    
    
15. Now reboot the vCenter server from the command line, using the command:
    
    reboot
    
    
    
16. The expected result after the reboot, would be that the Out of Sync message is no longer seen when looking at the Summary tab in the Networking view when focused on the vDS.  This confirms that the inconsistency on the VC database is resolved.
    
    
    • You can check this by logging into the vCenter after it comes up and all the services are started normally.  (vCenter --> networking icon View --> vDS --> Summary)
      
      
      
17. The last step is to consolidate (delete) the revert point snapshot(s) we took of the vCenter(s), as the revert point(s) will no longer be needed.  

• • This can be done from vCenter itself; no need to log into the host directly as before.
  • If the vCenter is part of a linked mode configuration, consolidate (delete) the revert point snapshot for each of the other vCenters. 
  • Do this as soon as possible, since vCenter tends to be a busy VM and the snapshot files would grow considerably large and quickly, which is not a desirable thing for multiple reasons. 

Additional Information

IMPORTANT NOTES:

1. It is important to follow each step in the sequence indicated.
2. Do not do any additional steps in the exercise.
3. Do not skip any steps.
4. If at any time you observe a result other than expected, STOP and do not attempt resolution.  Open a P1 Networking case with Broadcom using the instructions at Creating and managing Broadcom support cases