Resolving IP Conflict with vSAN File Services Internal Docker Network on vSphere vSwitch Virtual Machines.

Products

• VMware vSphere ESXi

Issue/Introduction

• This article addresses an IP address conflict scenario where a production subnet utilizing the ###.##.#.#/16 range overlaps with the default internal Docker network created by vSAN File Services appliances.
  
  
• This overlap can lead to routing issues and communication failures for file shares attempting to reach that production subnet.
  
  
• This document outlines the procedure to change the internal Docker network range before activating vSAN File Services to prevent such conflicts.

Environment

• VMware vCenter Server 8.x
• VMware vSphere ESXi

Cause

The core issue stems from the operating system's routing table. When Docker creates its internal bridge networks, these networks typically register routes on the host (in this case, the vSAN File Services appliances). 

If the internal Docker network range overlaps with an external network segment ###.##.#.#/16 production subnet, the operating system's routing table might prioritize the internal Docker route over the route to the external physical network interface. 

Consequently, traffic destined for the production ###.##.#.#/16 subnet is misrouted towards the internal Docker bridge instead of the physical network where the production subnet resides, leading to a "black hole" ( region where data packets are silently discarded without notifying the sender) for that traffic.

Resolution

To avoid this routing conflict, the internal Docker network range used by vSAN File Services appliances must be explicitly changed to a non-overlapping private IP range before activating vSAN File Services.

Additional Information

• What is the correct procedure to change the internal Docker network range during the vSAN File Services activation process?

• The procedure involves using the `configure-vsan-file-service` command-line utility on the vCenter Server Appliance (VCSA) *before* you enable vSAN File Services in the vCenter UI.
  

• Does the internal Docker network necessarily need to be a /16 network?
  • No, the internal Docker network does not necessarily need to be a /16. While /16 is a common default for Docker bridges, you can configure it to be a smaller subnet, such as a /24, /20, or any other appropriate private CIDR block, as long as it does not overlap with any of your existing production networks or other internal Docker networks.
    
    
    
• Can the internal network range of Docker using /usr/lib/vmware-vsan/bin/configure-vsan-file-service --set-container-network 0.0.0.0/0 before activating vSAN File Services?
  • Yes, the command /usr/lib/vmware-vsan/bin/configure-vsan-file-service is indeed the correct utility. However, you should NOT use 0.0.0.0/0 as the CIDR. 0.0.0.0/0 represents all IP addresses and would cause massive routing issues. You must specify a specific, non-overlapping private CIDR block that you have chosen for the internal Docker network (e.g., ##.###.#.#/16, ###.###.###.#/24).