vCenter reports "The static MAC address (04:50:56:##:##:##) of <vm1> conflicts with MAC assigned to <vm2>" in a VKS environment

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• vCenter is occasionally reporting duplicate MAC addresses in the environment:
  "The static MAC address (04:50:56:##:##:##) of <vm1> conflicts with MAC assigned to <vm2>"
• The MAC address in question does start with "04:50:56:", being specific to VKS
• The duplicates are between VKS-specific workload, such as vSphere Pods and Control Plane- and Worker-virtual machines.

Environment

VMware vCenter
VMware vSphere Kubernetes Service
VMware NSX

Cause

This is due to a small gap between vCenter and NSX: When a jobs are run on the Supervisor Cluster - which are created as vSphere Pods -, a corresponding available MAC address (with prefix 04:50:56:) is allocated from NSX Manager. When said pod shuts down and enters the Completed state, it cannot be started anymore and hence the reserved MAC address is freed from NSX Manager's available MAC address pool.

As such one-off pods remains on the environment by default and are still known to and registered in vCenter, any new workload using the previously freed MAC address will cause vCenter to detect a duplicate MAC address. Because however such pods cannot be started anymore, there is no "real" duplicate MAC address issue present on the network.

Resolution

Solution

This is a false-positive alert. Engineering is looking to improve this in the future.

Workaround

Finished jobs are not deleted as per Kubernetes default. To reduce likelihood of entities using the same MAC address from vCenter's perspective, finished jobs can be changed to be automatically deleted. For this, the setting .spec.ttlSecondsAfterFinished can be configured. (For more information, please refer to the Kubernetes documentation at https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/)

Additional Information

If needed it is possible to check for any duplicate MAC address assignments on NSX Manager, by either utilizing the log bundle or querying the NSX API.

1. Retrieve current MAC address assignments of NSX's Logical Ports
    
   
   1. When using a extracted log bundle from NSX Manager, following can be used:
      # cat ./nsx_manager_[UUID].tar_extracted/desired_state_manager.json | jq -r '."/nsxapi/api/v1/logical-ports"' > logicalports.json
        
   2. To query current information from NSX Manager's API:
      # curl -k -X GET -u "admin:<NSX-PASSWORD>" https://localhost/api/v1/logical-ports > logicalports.json
        
2. Check for any duplicate MAC address assignments
   # grep -o '"mac_address"\s*:\s*"[^"]*"' logicalports.json | cut -d'"' -f4 | sort | uniq -d
04:50:56:##:##:##
   
   Note: If there is no output, there was no duplicate MAC address found. If there is, duplicate MAC addresses will be listed.