vmk0 management network MAC address is not updated when NIC card is replaced or vmkernel has duplicate MAC address

Products

• VMware vSphere ESXi

Issue/Introduction

• The MAC address of vmk0 management network may not get updated after you replace or assign a new MAC address to the NIC and run the esxcfg-vmknic –l command from the command line. The default setting for the ESXi host is not to select the new hardware MAC address.
• The management network in inaccessible on an ESXi host.
• You are unable to ping vmkernel.
• You are unable to ping a decommissioned ESXi physical server that has been rebuilt.
• Under certain circumstances there may be a situation where a duplicate mac address arises on the network due to ESXi vmkernel maintaining the mac address of a previous ESXi installation on a different physical server.
• During initial install on the previous server vmkernel vmk0 interface will use the physical server's mac address assigned to one of the vmknic interfaces instead of a random 00:50:56::XX:XX:XX.
• On decommissioning the server if the installation USB or Local disk is moved to another server or copied if a LUN, the vmkernel vmk0 interface on the new server will still have the previous server's physical MAC address.

Environment

VMware vSphere ESXi 6.0

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Resolution

Duplicate MAC addresses

Confirm by checking the physical MAC address of all physical NICs and their link status, the existing vSwitch configuration, and the current vmkernel interfaces configuration. For ESXi, use these commands:

# esxcfg-nics -l

# esxcfg-vswitch -l
# esxcfg-vmknic -l

If a duplicate MAC address is confirmed, the MAC address must be changed by deleting and recreating the vmkernel interface.

 

To delete a vmknic from a port group, use this command:

# esxcfg-vmknic -d -p pgName

or

# esxcfg-vmknic -d pgName

To add a vmknic to a port group, run the command:

# esxcfg-vmknic -a -i DHCP -p pgName

or

# esxcfg-vmknic -a -i x.x.x.x -n 255.255.255.0 pgName

 

Workaround

 

To work around the issue, manually configure the MAC address on the ESXi host:

1. In the troubleshooting console, run the command:
   
   esxcfg-advcfg -s 1 /Net/FollowHardwareMac
    
2. Restart the ESXi server.

Additional Information

If the /Net/FollowHardwareMac option is not being honored after a host reboot on ESXi 7.x, ensure sure the mac_mode of vmk0 is set to PNIC_BASED in the ESXi host configstore.

The steps to check and properly update the configstore are:
1) Export current configstore vmknic settings:
# configstorecli config current get -c esx -g network -k vmknics >> /tmp/My-Net.json
2) Remove the existing "mac" line and modify the "mac_mode" and "nic" settings in the file for vmk0:
# vi /tmp/My-Net.json  <---- Delete the current "mac" line, then add/modify the two lines below: 
   "mac_mode": "PNIC_BASED",
   "nic": "vmnic0",
# cat /tmp/My-Net.json (verify)

Example: 
[root@vESXi70-2:~] cat /tmp/My-Net.json
[
  {
   "device": "vmk0",
   "enabled": true,
   "enabled_services": {
     "management": true
   },
   "ip": {
     "dhcp": false,
     "dhcp_dns": false,
     "ipv4_address": "10.0.xx.yy",
     "ipv4_subnet_mask": "255.255.255.0",
     "ipv6": {
      "auto_configuration_enabled": true,
      "dhcp": false,
      "dhcp_dns": false
     },
     "ipv6_enabled": true
   },
   "ip_route_default_gateway": "10.0.xx.yy",
   "ip_route_ip_v6_default_gateway": "::",
   "mac_mode": "PNIC_BASED", <------------------ modify this line  
   "nic": "vmnic0",              <------------------ added this line
   "net_stack_instance_key": "defaultTcpipStack",
   "port_connection_type": "VSS_PORT_GROUP",
   "port_group": "VMkernel",
   "tso_mss": 0
  }
]
3) Create a backup copy of the Config-store, just in case: 
# cp /tmp/My-Net.json /My-Net.json
4) Import the changes made to /tmp/My-Net.json into the configstore:
# configstorecli config current set -c esx -g network -k vmknics -infile /tmp/My-Net.json
5) Confirm the changes made to the configstore:
# configstorecli config current get -c esx -g network -k vmknics
 

 

Creating an Alarm to Monitor Duplicate IP address: This method can be applied to warn of duplicate MAC addresses as well.

In addition to logging to the VMkernel logs, ESXi also logs this observation in /var/log/vobd.log file which stands for the VMkernel Observation. These observations can provide critical identifying information in case of an error and is usually used during troubleshooting. In our case, if we are seeing an intermittent network connectivity to our ESXi host which is in result of a duplicate IP Address. Utilizing these VOBs is that you can create vCenter Alarms when a specific VOB has been detected.

You can do exactly the same for detecting a duplicate IP Address for an ESXi host. First, identify the VOB ID by looking in the /var/log/vobd.log file (for Duplicate IP address or Duplicate MAC address):

YYYY-MM-DDTHH:MM:SS.513Z: [netCorrelator] 917174784727us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for xx.xx.xx.xx on interface vmk0, current owner being xx:xx:xx:xx:xx:xx

Note:

• The VOB ID for this is esx.problem.net.vmknic.ip.duplicate (in the case of a duplicate IP address) or VmMacConflictEvent or VmStaticMacConflictEvent (in the case of a conflicted MAC address) and this will be used in the vCenter Alarm trigger.
• The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
   

To create an alarm:

1. Create a new Alarm and specify a name, the Monitor type will be Hosts and Monitor for a specific event.
2. Copy the VOB ID identified above and specify that as the alarm Trigger.
3. If you want to receive an email notification or send an SNMP trap, configure additional actions, or click Next which displays a vCenter Server alert in the UI.
4. To confirm the alarm trigger, assign a test virtual machine with the IP Address/MAC address of an ESXi host to this virtual machine, the alarm now appears in the tasks/events.