Network outage for virtual machines on NSX segments when physical switch is rebooted

Products

• VMware NSX

Issue/Introduction

• Physical switches are rebooted.
• Virtual Machines on the ESXi hosts connected to the virtual machines are impacted.
• The virtual machine logs may indicate connectivity issues to devices such as NFS:

<DATE/TIME> <SERVER> kernel: ... nfs: server <nfs server> not responding, still trying

• On the ESXi host /var/run/log/vobd.log we see the vmnic of the ESXi host going down and backup later:

<DATE/TIME> In(14) vobd [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down
<DATE/TIME> In(14) vobd [vob.net.vmnic.linkstate.down] vmnic vmnic2 linkstate down
<DATE/TIME> In(14) vobd [vob.net.dvport.uplink.transition.down] Uplink: vmnic0 is down. 
<DATE/TIME> In(14) vobd [vob.net.vmnic.linkstate.up] vmnic vmnic2 linkstate up
<DATE/TIME> In(14) vobd [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
<DATE/TIME> In(14) vobd [vob.net.vmnic.linkstate.up] vmnic vmnic0 linkstate up

• The time match the times of the physical switch reboots.

Cause

Virtual machines loss connection due to the physical network being disconnected on the ESXi host, during the switch reboot.

Resolution

Reboot physical switches during a maintenance window and place ESXi hosts connected to the physical switch in maintenance mode prior to the reboot.