Connectivity Issues Between Cloud Proxy and Connected Nodes Due to Network Overlap

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VCF Operations

Issue/Introduction

Cloud Proxy nodes may experience connectivity issues if there is an overlap between the Docker and the external environment network.

Cases:

• Upgrade to Cloud Proxy version 8.16 and above
  
  
• New deployment of Cloud Proxy version 8.16 and above
  
  

Symptoms:

• Analytics nodes become unreachable
  
  
• Monitoring endpoints become unreachable
  
  
• Other network destinations become unreachable

Environment

VCF Operations: 9.1

VCF Operations Cloud Proxy 9.1.1

VMware Aria Operation: 8.16 and above

Cause

Starting from Cloud Proxy version 8.16, user-defined bridge networks have been introduced within Cloud Proxy.

• The 172.17.0.0/16 subnet is used for the default bridge network.
  
  
• For user-defined bridge networks, Docker assigns a different subnet by incrementing the second byte (e.g., 172.18.0.0/16).
  
  

If a Docker network overlaps with the external environment network, connectivity problems may occur, as network packets won't be routed outside the Cloud Proxy but will instead be routed internally.

You can use the tracepath <host-address> command to check the route towards the destination host.

Note: there may be multiple containers based on the Cloud Proxy’s type and configuration. 

Resolution

Currently, there is no permanent resolution to update the docker bridge network pool.

Workaround:

Docker does not provide a direct way to modify the subnet of an existing user-defined network without deleting and recreating it. Docker's user-defined network configuration is set at the time the network is created, and there is no built-in command to change the subnet of an active network.
The standard workaround is to remove the existing network and create a new one with the desired subnet configuration.

Steps for Workaround:

 

1. To find the Docker networks, run:
   

   docker network ls

   Keep the network NAME for further use.

2. To find the containers in the given network, run:
   

   docker network inspect <network_name>

    Under the Containers field, you’ll see a list of all containers connected to the specified network.

3. Stop all running containers on the network you want to update the subnet for:
   

   docker stop <container_name>

4. Disconnect containers from the network:
   

   docker network disconnect <network_name> <container_name>

5. Remove the existing custom network:
   

   docker network rm <network_name>

6. Recreate the network with the desired subnet configuration and specify a gateway based on the subnet, using the same network name:
   

   docker network create --subnet=<new_subnet> --gateway <gateway> <network_name>

7. Reconnect containers to the newly created network:
   

   docker network connect <network_name> <container_name>

8. Start the containers again:
   

   docker start <container_name>