Network communication failed on a VM network adapter after a vmotion due to an attempted change in MAC address by the VM

Products

• VMware vSphere ESXi

Issue/Introduction

• There was a vmotion of a VM. After that network communication to the VM failed on one of the network adapters.
• The VM is running a version of a Linux OS.
• You see logs similar to the following in the vmkernel log on the destination host in /var/run/log that indicate the port the network adapter was connecting to was blocked by a security policy during the migration:

In(182) vmkernel: cpu5:10460142)cswitch: L2Sec_EnforcePortCompliance:374: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client <VM Name>.eth2 has policy violations on port <PORT ID>. Port is blocked

• In the same vmkernel log you see similar logs to the following which indicate it was blocked as it was trying to change its MAC address: 

In(182) vmkernel: cpu5:10460142)cswitch: L2Sec_EnforcePortCompliance:237: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client <VM Name>.eth2 requested mac address change to XX:XX:XX:XX:XX:1b on port <PORT ID>, disallowed by vswitch policy

• It is blocked since the security policy on the vswitch is to reject MAC address changes.
• When you check the vmx file or in the vCenter UI you can see that the automatically generated MAC address is different to the MAC address referenced in the log message about the "mac address change". 
• If you run the 'ip addr' command on the guest OS, then you get output similar to the following. It can be seen that the MAC address being referenced in the logs, which in this example is "XX:XX:XX:XX:XX:1b", is the MAC address assigned to the adapter by the guest OS. But there is a reference to the permaddr, which in this example is  XX:XX:XX:XX:XX:b2 , and that is the MAC address found in the vmx file:

ip a for eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether XX:XX:XX:XX:XX:1b brd ff:ff:ff:ff:ff:ff permaddr XX:XX:XX:XX:XX:b2

Environment

VMware vSphere ESX

Cause

• The port is being blocked since there is a mismatch between the vmx generated MAC address and the configured MAC address on the guest OS level. Due to this it is being blocked since the security policy on the vswitch is to reject MAC address changes.
• The permaddr is being reported by Linux since this is a reference to the original MAC address assigned to the interface. The original MAC address is the one which was automatically generated by vmx at the time the adapter was first created.
• Linux is assigning the incorrect MAC address XX:XX:XX:XX:XX:1b since it has been incorrectly configured in a guest OS config file. The setting in the config file that matters is the option MACADDR and in most Linux distributions it is usually found in /etc/sysconfig/network-scripts directory.
• Below is an example of a config file with the incorrectly configured option "MACADDR=XX:XX:XX:XX:XX:1b":

[root@<VM Name> network-scripts]# cat ifcfg-eth2
DEVICE=eth2
MACADDR=XX:XX:XX:XX:XX:1b
NAME=eth2
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPADDR=X.X.X.X
GATEWAY=X.X.X.X
PREFIX=24

Resolution

• Correcting or removing the MACADDR option will stop what is perceived as an attempted MAC address change by L2 security on the vswitch, hence preventing the port from being blocked when it is migrated.
• Remove the MACADDR option from the guest OS network interface configuration file if not needed.
• Another solution is to correct MACADDR option in the config file so it matches with the original vmx generated MAC address. Using the example above the value for MACADDR would be corrected from XX:XX:XX:XX:XX:1b to XX:XX:XX:XX:XX:b2.

Additional Information

• You can find the automatically generated MAC address in the vCenter UI by checking in VM -> Edit Settings -> Virtual Hardware and expand the Network Adapter setting.
• Also you can find the automatically generated MAC address by running a grep like the following against the vmx file:
  
  - grep -i generatedAddress <VM Name>.vmx