vCenter network connectivity lost - Recover vCenter network when connected to a Distributed Switch

Products

• VMware vCenter Server

Issue/Introduction

This guide will help you build a temporary Standard Switch where you will connect vCenter to recover from the network down scenario. Then you can make the necessary changes in the DVS to return to a good state. 

Most of the steps are only possible via command line.

Symptoms:

1. Management Network only exists in a Distributed Switch
2. There are no Ephemeral Ports in the cluster
3. vCenter Server lost network connectivity after an unplanned or planned outage.
4. You cannot connect vCenter back to a Distributed Switch portgroup in the same or different hosts
5. You are unable to open the vCenter vSphere Client to make any changes to the network because vCenter network connection is down
6. This error appears when trying to modify network settings on any ESXi host. Also, it can show up if you want to change network adapters for a ESXi host connected to a DVS with non-ephemeral ports:

Addition or reconfiguration of network adapters attached to non-ephemeral distributed virtual port groups is not supported.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

If vCenter is connected to a Distributed Switch and it loses network access, vCenter is not able to connect to a Distributed Port because it does not have access to the ESXis.

VMware recommends that you configure Ephemeral Ports for Management Network in your environment to avoid this issue from happening again. For more information: Static (non-ephemeral) or ephemeral port binding on a vSphere Distributed Switch (1022312)

***Please note if you have LACP configured on your physical switch you will need to temporarily break the LACP configuration so you can remove a NIC from the vDS to be available for the vCenter to use on the standard switch (unless you have non-LACP NICs available and configured to pass the traffic). If you do not break the LACP configuration before moving the NIC off the vDS this can cause further issues to your environments connectivity***

***Please also ensure before proceeding you have access to the DCUI/iLO/iDRAC for the host you'll be performing the below steps***

Resolution

Follow this Step-by-Step guide:

1. Remove a vmnic located in the DVS connected to the Management Network
   
   
   1. Identify the Port ID where the vmnic that you want to remove is connected to the DVS

       

      esxcli network vswitch dvs vmware list | egrep "Client: vmnic#" -A3

      Sample output

      esxcli network vswitch dvs vmware list | egrep "Client: vmnic1" -A3
               Client: vmnic1
               DVPortgroup ID: dvportgroup-5008
               In Use: true
               Port ID: 12

   2. Remove the vmnic

      esxcfg-vswitch -Q vmnic# -V PortID DVSName

      Example using vmnic1, Port ID 12 and DVS Name ProdSwitchDVS:

      esxcfg-vswitch -Q vmnic1 -V 12 ProdSwitchDVS

       

2. Create a Standard Switch, a Portgroup, add the vmnic to the Standard Switch
   
   
   1. Create a Standard switch

       

      esxcli network vswitch standard add --vswitch-name=vSwitchName

   2. Create a Portgroup

      esxcli network vswitch standard portgroup add --portgroup-name=PortgroupName --vswitch-name=vSwitchName

   3. Add a vmnic to the Standard Switch

       

      esxcli network vswitch standard uplink add --uplink-name=vmnic --vswitch-name=vSwitchName

       

3. Recover vCenter’s Virtual Machine network connectivity
   
   First, we will connect vCenter’s virtual machine to the new Standard Switch Portgroup. This will help to regain network access to vCenter. This will also allow the ESXIs to connect back to vCenter Server too and you will be able to manage your infrastructure again.
   
   
   1. Login to the ESXi vSphere Client with administrator credentials
   2. Go to “Virtual Machines”
   3. Check vCenter’s Virtual Machine
   4. Click in “Actions” > “Edit Settings”
   5. Connect Network Adapter 1 to the newly created Standard Switch Portgroup
   6. Click Save
      
      Note: Up until now you should have recovered vCenter’s network connectivity and now you should be able to connect to its vSphere Client. If you still can’t, make sure the Standard Switch Portgroup has the correct VLAN and MTU configuration
   
   Once you have reviewed that you can see everything is ok in your vCenter Inventory, let’s now migrate vCenter back to the DVS so that we can have the same configuration before the outage.
   
   
   1. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
   2. Go to the vCenter’s virtual machine, right click it and select “Edit Settings”.
   3. Connect Network Adapter 1 to the Management Distributed Switch Portgroup
   4. Click OK
      
      Make sure you do not lose network access again. If everything is ok after a couple of minutes, continue to the next steps.
   
   
4. Migrate vmnic back to the DVS.
   
   Let’s put the vmnic back to the DVS by following these steps:
   
   
   1. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
   2. Go to the vCenter’s virtual machine, right click it and select “Edit Settings”.
   3. Connect Network Adapter 1 to the Management Distributed Switch Portgroup
   4. Click OK
   5. Make sure you do not lose network access again. If everything is ok after a couple of minutes, continue to the next steps.
   6. Migrate the vmnic and vmk back to the DVS
   7. Let’s put the vmnic back to the DVS by following these steps:
   8. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
   9. Go to the “Networking” tab.
   10. Right click the DVS and select “Add and Manage Hosts”
   11. Select “Manage host networking” and click Next
   12. Click “Attached hosts…”
   13. Check the ESXi with the vmk and vmnic that you want to add back to the DVS and click OK
   14. Click Next
   15. On the "Manage physical adapters" list, select the vmnic and click “Assign uplink”
   16. Select an Uplink with empty “Assigned Adapter” and click OK
   17. Click Next
   18. Click Next in "Manage VMkernel adapters"
   19. Click Next in "Migrate VM networking"
   20. Click Finish
5. Delete the Standard Switch.
   
   You are now able to delete the temporary Standard Switch we created to recover vCenter’s network connectivity.
   
   
   1. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
   2. Go to the “Hosts and Clusters” Tab
   3. Select the ESXi used during this process
   4. Click in the “Configure” tab
   5. Click on “Virtual Switches” under “Networking”
   6. Look for the temporary Standard Switch you created and click the ellipsis “…”
   7. Click “Remove”
   8. Click “Yes” in the warning that pops up
      
      
      

You have now successfully recovered vCenter’s network connectivity and the environment should be up and running.

Additional Information

Static (non-ephemeral) or ephemeral port binding on a vSphere Distributed Switch (1022312)
Configuring vSwitch or vNetwork Distributed Switch from the command line in ESXi/ESX (1008127)

Impact/Risks:
There should be at least 2 vmnics used for the Management Network because in one of the steps we will remove one vmnic from the DVS Management Portgroup so that we can use it for the Standard Switch that we will create temporarily.

NOTE: If the vmnics are in an LACP configuration, that will need to be broken on the physical switch to avoid downtime. Follow this KB Enable EtherChannel / Link Aggregation Control Protocol (LACP) in ESXi/vCenter (broadcom.com) for steps on how to work with an LACP configuration.

If you do not have 2 vmnics in the ESXi, it is recommended that you follow these steps via DCUI Shell. Otherwise, you will lose access to SSH when you run the remove vmnic command and won’t be able to continue with the process.