Network connectivity loss for multiple VMs on a shared VLAN across multiple ESXi hosts.
search cancel

Network connectivity loss for multiple VMs on a shared VLAN across multiple ESXi hosts.

book

Article ID: 437375

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

  • Multiple Virtual Machines (VMs) hosted across several ESXi hosts within a vSphere environment experienced a total loss of network connectivity on <VLAN_ID>.

  • Analysis of hostd.log across multiple affected ESXi hosts (var/run/log/hostd.log) correlate these network disruptions to tasks initiated from vCenter Server, as evidenced by the VC opID mapping to vmkernel operations.

    In(182) vmkernel: cpu0:2099939 opID=46###c50)World: 12##0: VC opID mmx2m1e7-####-auto-aete-h5:70033639-###-c4ef maps to vmkernel opID 46c###c50
    In(182) vmkernel: cpu0:2099939 opID=46###c50)NetPort: 3##0: blocking traffic on DV port XX
    vpxd[43##0]: Event [1366###3] [1-1] [vim.event.DvsPortBlockedEvent] [info] [] [Vcenter-ID] [136###03] [The dvPort XX was blocked in the vSphere Distributed Switch <VDS_NAME>. It was in Unblocked state before.

  • Checking the Vcenter journalctl (commands/journalctl) logs could see that there was a port group level block was initiated by the admin account locally (VSPHERE.LOCAL\Administrator):

    [vim.event.DVPortgroupReconfiguredEvent] [info] [VSPHERE.LOCAL\Administrator] [Vcenter-ID ] [13###75] [dvPort group <DV_Port-Group> in "Vcenter-Name" was reconfigured.

                  Modified:

                  config.defaultPortConfig.blocked.value: false -> true;

                  Added:

                  Deleted:

                  ]

  • This configuration change results in the immediate blocking of all ports within the affected Distributed Port Group. Log analysis confirms that each port associated with the group transitioned to a blocked state, as indicated by the following event signatures from the journalctl (commands/journalctl) logs on the Vcenter:

    [vim.event.DvsPortblockedEvent] [info] [] [Vcenter_Name] [13###516] [The dvPort XX was blocked in the vSphere Distributed Switch <VDS_NAME> in <Vcneter_Name>. It was in Unblocked state before.]
    [vim.event.DvsPortblockedEvent] [info] [] [Vcenter_Name] [13###517] [The dvPort YY was blocked in the vSphere Distributed Switch <VDS_NAME> in <Vcneter_Name>. It was in Unblocked state before.]
    [vim.event.DvsPortblockedEvent] [info] [] [Vcenter_Name] [13###518] [The dvPort ZZ was blocked in the vSphere Distributed Switch <VDS_NAME> in <Vcneter_Name>. It was in Unblocked state before.]

 

Environment

VMware vCenter Server

VMware ESXI Hosts

Cause

  • The network outage was caused by an administrative configuration change rather than an underlying physical or virtual network failure. Specifically, a manual modification was performed via the vCenter Server UI by the VSPHERE.LOCAL\Administrator account on the affected Distributed Port Group.
  • During this administrative action, the config.defaultPortConfig.blocked.value attribute was toggled from false to true. This resulted in an immediate, explicit block of all Distributed Virtual Ports (dvPorts) associated with the group, leading to a total loss of network connectivity for all resident virtual machines.

Resolution

To restore connectivity, the Distributed Port Group configuration must be reverted to an unblocked state.

  1. Log in to the vSphere Client.

  2. Navigate to Inventory > Networking.

  3. Select the affected Distributed Virtual Switch and locate the impacted Distributed Port Group.

  4. Right-click the port group and select Edit Settings.

  5. Navigate to the Miscellaneous section.

  6. Locate the Block All Ports option.

  7. Set the value to No (ensure the checkbox is deselected) to unblock the ports.

  8. Click OK to save the changes.

  9. Verify that connectivity is restored to the impacted VMs and that vim.event.DvsPortUnblockedEvent entries appear in the vCenter Tasks and Events

Additional Information

Powered by Wolken Software