Network connectivity loss for multiple VMs on a shared VLAN across multiple ESXi hosts.
Article ID: 437375
Updated On:
Products
Issue/Introduction
- Multiple Virtual Machines (VMs) hosted across several ESXi hosts within a vSphere environment experienced a total loss of network connectivity on <VLAN_ID>.
- Analysis of hostd.log across multiple affected ESXi hosts (var/run/log/hostd.log) correlate these network disruptions to tasks initiated from vCenter Server, as evidenced by the VC opID mapping to vmkernel operations.
In(182) vmkernel: cpu0:2099939 opID=46###c50)World: 12##0: VC opID mmx2m1e7-####-auto-aete-h5:70033639-###-c4ef maps to vmkernel opID 46c###c50In(182) vmkernel: cpu0:2099939 opID=46###c50)NetPort: 3##0: blocking traffic on DV port XXvpxd[43##0]: Event [1366###3] [1-1] [vim.event.DvsPortBlockedEvent] [info] [] [Vcenter-ID] [136###03] [The dvPort XX was blocked in the vSphere Distributed Switch <VDS_NAME>. It was in Unblocked state before. - Checking the Vcenter journalctl (commands/journalctl) logs could see that there was a port group level block was initiated by the admin account locally (VSPHERE.LOCAL\Administrator):
[vim.event.DVPortgroupReconfiguredEvent] [info] [VSPHERE.LOCAL\Administrator] [Vcenter-ID ] [13###75] [dvPort group <DV_Port-Group> in "Vcenter-Name" was reconfigured.Modified:config.defaultPortConfig.blocked.value: false -> true;Added:Deleted:] - This configuration change results in the immediate blocking of all ports within the affected Distributed Port Group. Log analysis confirms that each port associated with the group transitioned to a blocked state, as indicated by the following event signatures from the journalctl (commands/journalctl) logs on the Vcenter:
[vim.event.DvsPortblockedEvent] [info] [] [Vcenter_Name] [13###516] [The dvPort XX was blocked in the vSphere Distributed Switch <VDS_NAME> in <Vcneter_Name>. It was in Unblocked state before.][vim.event.DvsPortblockedEvent] [info] [] [Vcenter_Name] [13###517] [The dvPort YY was blocked in the vSphere Distributed Switch <VDS_NAME> in <Vcneter_Name>. It was in Unblocked state before.][vim.event.DvsPortblockedEvent] [info] [] [Vcenter_Name] [13###518] [The dvPort ZZ was blocked in the vSphere Distributed Switch <VDS_NAME> in <Vcneter_Name>. It was in Unblocked state before.]
Environment
VMware vCenter Server
VMware ESXI Hosts
Cause
- The network outage was caused by an administrative configuration change rather than an underlying physical or virtual network failure. Specifically, a manual modification was performed via the vCenter Server UI by the VSPHERE.LOCAL\Administrator account on the affected Distributed Port Group.
- During this administrative action, the config.defaultPortConfig.blocked.value attribute was toggled from false to true. This resulted in an immediate, explicit block of all Distributed Virtual Ports (dvPorts) associated with the group, leading to a total loss of network connectivity for all resident virtual machines.
Resolution
To restore connectivity, the Distributed Port Group configuration must be reverted to an unblocked state.
Log in to the vSphere Client.
Navigate to Inventory > Networking.
Select the affected Distributed Virtual Switch and locate the impacted Distributed Port Group.
Right-click the port group and select Edit Settings.
Navigate to the Miscellaneous section.
Locate the Block All Ports option.
Set the value to No (ensure the checkbox is deselected) to unblock the ports.
Click OK to save the changes.
Verify that connectivity is restored to the impacted VMs and that
vim.event.DvsPortUnblockedEvententries appear in the vCenter Tasks and Events
Additional Information
- Please refer to the kb if NSX ports are blocked : VM port blocked state on NSX segment prevents connectivity
- Please refer to the kb if port are blocked due to security policy mismatch: vSphere Distributed Switch Ports Block Traffic Due to Security Policy Mismatch in vCenter
Feedback
