vCenter network connectivity lost - Recover vCenter network when connected to a Distributed Switch

search cancel

vCenter network connectivity lost - Recover vCenter network when connected to a Distributed Switch

book

Article ID: 318719

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This guide will help you build a temporary Standard Switch (vSS) where you will connect vCenter VM to recover from a port disconnection/network down scenario.
When vCenter is online again, you can make the necessary changes in the distributed switch (vDS) and reconnect vCenter there.
Most of the steps are only possible via command line.

Management Network only exists in a Distributed Switch
There are no Ephemeral Ports in the cluster
vCenter Server lost network connectivity after an unplanned or planned outage
You cannot connect vCenter back to a Distributed Switch portgroup in the same host, or if registered on different hosts
You are unable to open the vCenter vSphere Client to make any changes to the network because vCenter network connection is down
vCenter and vSAN on vDS (vCenter VM on vSAN Datastore) with change to config resulting in vSAN datastores going offline. (standard switches have to be built for both MGT and vSAN with VLANs)
The below error appears when trying to modify vDS port adapter settings on any ESXi host. Also, it can show up if you want to change network adapters for a ESXi host connected to a vDS with non-ephemeral ports:

Addition or reconfiguration of network adapters attached to non-ephemeral distributed virtual port groups is not supported.

Cause

If vCenter to host communication is lost, VMs will not be able to be reconfigured to static (also known as non-ephemeral) port groups on the vDS because vCenter is unavailable to give the VM a port binding.

VMware recommends that you configure an Ephemeral Port Binding dvportgroup for your VC's management network in your environment to avoid this issue from happening again.
For more information, see: Static (non-ephemeral) or ephemeral port binding on a vSphere Distributed Switch.

***Please note: if you have LACP configured on your physical switch, you will need to temporarily break the LACP configuration so you can remove a NIC from the vDS; in order to be available for the vCenter VM to use on the standard switch (unless you have non-LACP NICs available and configured to pass the traffic). If you do not break the LACP configuration before moving the NIC off the vDS, this can cause further issues to your environment's connectivity***

***Before proceeding, please also ensure you have access to the DCUI/iLO/iDRAC for the host you'll be performing the below steps***

Resolution

Remove a vmnic located in the vDS managing the vCenter VM's network / VLAN:
1. Identify the Port ID#, vmnic# and vDSName where the vmnic that you want to remove is connected to the vDS
  esxcli network vswitch dvs vmware list
  
  Sample output:
  Name: vDSName VDS ID: ######## Class: vswitch Num Ports: #### Used Ports: ## Configured Ports: ## MTU: 9000/1500 CDP Status: listen Beacon Timeout: -/+# Uplinks: vmnic#, vmnic# VMware Branded: true DVPort: Client: vmnic# DVPortgroup ID: dvportgroup-### In Use: true Port ID: ##
2. Remove the vmnic
  esxcfg-vswitch -Q vmnic# -V PortID# vDSName
  
  Example using vmnic1, Port ID 12 and vDS Name ProdSwitchvDS:
  esxcfg-vswitch -Q vmnic1 -V 12 ProdSwitchvDS
Create a Standard Switch, a Portgroup, add the vmnic to the Standard Switch
1. Create a Standard switch
  esxcli network vswitch standard add --vswitch-name=vSwitchName
2. Create a Portgroup
  esxcli network vswitch standard portgroup add --portgroup-name=PortgroupName --vswitch-name=vSwitchName
3. Add a vmnic to the Standard Switch
  esxcli network vswitch standard uplink add --uplink-name=vmnic --vswitch-name=vSwitchName
Recover vCenter’s Virtual Machine network connectivity

First, we will connect vCenter’s virtual machine to the new Standard Switch Portgroup. This will help to regain network access to vCenter. This will also allow the ESXi hosts to connect back to vCenter Server too, and you will be able to manage your infrastructure again.
1. Login to the ESXi vSphere Client with administrator credentials
2. Go to “Virtual Machines”
3. Check vCenter’s Virtual Machine
4. Click in “Actions” > “Edit Settings”
5. Connect Network Adapter 1 to the newly created Standard Switch Portgroup
6. Click Save
  
  Note: Up until now you should have recovered vCenter’s network connectivity and now you should be able to connect to its vSphere Client. If you still can’t, make sure the Standard Switch Portgroup has the correct VLAN and MTU configuration
Once you have reviewed that you can see everything is ok in your vCenter Inventory, let’s now migrate vCenter back to the vDS so that we can have the same configuration before the outage.
1. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
2. Go to the vCenter’s virtual machine, right click it and select “Edit Settings”.
3. Connect Network Adapter 1 to the Management Distributed Switch Portgroup
4. Click OK
  
  Make sure you do not lose network access again. If everything is ok after a couple of minutes, continue to the next steps.
Migrate vmnic back to the vDS.

Put the vmnic back to the vDS by following these steps:
1. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
2. Go to the vCenter’s virtual machine, right click it and select “Edit Settings”.
3. Connect Network Adapter 1 to the Management Distributed Switch Portgroup
4. Click OK
5. Make sure you do not lose network access again. If everything is ok after a couple of minutes, continue to the next steps.
6. Migrate the vmnic and vmk back to the vDS
  1. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
  2. Go to the “Networking” tab.
  3. Right click the vDS and select “Add and Manage Hosts”
  4. Select “Manage host networking” and click Next
  5. Click “Attached hosts…”
  6. Check the ESXi with the vmk and vmnic that you want to add back to the vDS and click OK
  7. Click Next
  8. On the "Manage physical adapters" list, select the vmnic and click “Assign uplink”
  9. Select an Uplink with empty “Assigned Adapter” and click OK
  10. Click Next
  11. Click Next in "Manage VMkernel adapters"
  12. Click Next in "Migrate VM networking"
  13. Click Finish
Delete the Standard Switch.

You are now able to delete the temporary Standard Switch we created to recover vCenter’s network connectivity.
1. If you are not logged in the vCenter vSphere Client already, login to vCenter with Administrator credentials.
2. Go to the “Hosts and Clusters” Tab
3. Select the ESXi used during this process
4. Click in the “Configure” tab
5. Click on “Virtual Switches” under “Networking”
6. Look for the temporary Standard Switch you created and click the ellipsis “…”
7. Click “Remove”
8. Click “Yes” in the warning that pops up

You have now successfully recovered vCenter’s network connectivity and the environment should be up and running.

Additional Information

Static (non-ephemeral) or ephemeral port binding on a vSphere Distributed Switch
Configuring vSwitch or vNetwork Distributed Switch from the command line in ESXi

Impact/Risks:
There should be at least 2 vmnics used for the Management Network because in one of the steps we will remove one vmnic from the vDS Management Portgroup so that we can use it for the Standard Switch that will be create temporarily.

NOTE: If the vmnics are in an LACP configuration, that will need to be broken on the physical switch to avoid downtime. Follow this KB Enable EtherChannel / Link Aggregation Control Protocol (LACP) in ESXi/vCenter for steps on how to work with an LACP configuration.

If you have less than 2 vmnics in the vDS, it is recommended that you follow these steps via DCUI Shell. Otherwise, you will lose access to SSH when you run the remove vmnic command and won’t be able to continue with the process.

Feedback

thumb_up Yes

thumb_down No