VM network adapter blocked due to guest OS MAC address change or mismatch following a vmotion or reboot

Products

• VMware vSphere ESXi

Issue/Introduction

• There was a vmotion or a reboot of a VM. After that network communication to the VM failed on one of the network adapters.
• The VM is running a version of a Linux OS.
• You see logs similar to the following in the vmkernel log on the destination host in /var/run/log that indicate the port the network adapter was connecting to was blocked by a security policy during the migration:

In(182) vmkernel: cpu5:10460142)cswitch: L2Sec_EnforcePortCompliance:374: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client <VM Name>.eth2 has policy violations on port <PORT ID>. Port is blocked

• In the same vmkernel log you see similar logs to the following which indicate it was blocked as it was trying to change its MAC address: 

In(182) vmkernel: cpu5:10460142)cswitch: L2Sec_EnforcePortCompliance:237: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client <VM Name>.eth2 requested mac address change to XX:XX:XX:XX:XX:1b on port <PORT ID>, disallowed by vswitch policy

• The security policy on the vswitch is to reject MAC address changes.
• When you check the vmx file or in the vCenter UI you can see that the automatically generated MAC address is different to the MAC address referenced in the log message about the "mac address change". 
• If you run the 'ip addr' command on the guest OS, then you get output similar to the following. It can be seen that the MAC address being referenced in the logs, which in this example is "XX:XX:XX:XX:XX:1b", is the MAC address assigned to the adapter by the guest OS. But there is a reference to the permaddr, which in this example is  XX:XX:XX:XX:XX:b2 , and that is the MAC address found in the vmx file:

ip a for eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether XX:XX:XX:XX:XX:1b brd ff:ff:ff:ff:ff:ff permaddr XX:XX:XX:XX:XX:b2

Environment

VMware vSphere ESX

Cause

• A mismatch between the Guest OS MAC address and the .vmx configuration triggers a violation of the 'MAC Address Changes' security policy on the virtual switch port."
• The permaddr is being reported by Linux since this is a reference to the original MAC address assigned to the interface. The original MAC address is the one which was automatically generated by VMware vmx at the time the adapter was first created.
• Linux is assigning the incorrect MAC address XX:XX:XX:XX:XX:1b since it has been incorrectly configured in a guest OS config file.
• The setting in the config file that matters is the option MACADDR and in most Linux distributions it is usually found in /etc/sysconfig/network-scripts directory.
• Below is an example of a config file with the incorrectly configured option "MACADDR=XX:XX:XX:XX:XX:1b":

[root@<VM Name> network-scripts]# cat ifcfg-eth2
DEVICE=eth2
MACADDR=XX:XX:XX:XX:XX:1b
NAME=eth2
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPADDR=X.X.X.X
GATEWAY=X.X.X.X
PREFIX=24

Resolution

• Correcting or removing the MACADDR option will stop what is perceived as an attempted MAC address change by L2 security on the vswitch, hence preventing the port from being blocked when it is migrated.
• • • • Steps to remove the MACADDR option from the guest OS network interface configuration file if not needed.
        
        1. From the guest OS CLI open the interface configuration file in /etc/sysconfig/network-scripts using a text editor.
        2. Delete the MACADDR line and save the changes.
        3. Restart the network service. 
           
           
      • Steps to correct MACADDR option in the config file so it matches with the original vmx generated MAC address.
        1. From the guest OS CLI open the interface configuration file in /etc/sysconfig/network-scripts using a text editor.
        2. Edit the value in the MACADDR line to match with the VMware vmx generated MAC address and save the changes. Using the example above the value for MACADDR would be corrected from XX:XX:XX:XX:XX:1b to XX:XX:XX:XX:XX:b2.
        3. Restart the network service.

Additional Information

• You can find the automatically generated MAC address in the vCenter UI by checking in VM -> Edit Settings -> Virtual Hardware and expand the Network Adapter setting.
• Also you can find the automatically generated MAC address by running a grep like the following against the vmx file:
  - grep -i generatedAddress <VM Name>.vmx
• vSwitch security and the MAC address change feature Technical Documentation.