Changing Docker Network in VMWare Cloud Gateway

Products

• VMware vCenter Server

Issue/Introduction

This article will help to mitigate the issue for customers, with management networks conflicting with default docker network used by VMWare Cloud Gateway.

Symptoms:

VMWare Cloud Gateway UI is not loading on browser due to network conflict.

Environment

VMware vCenter Server 7.0.x

Cause

VMWare Cloud Gateway uses Docker Engine to host the VMWare Agent Platform(VAP). Docker by default uses 172.17.0.0/16 as the default bridge Network. It also creates and new bridge network that by default uses the next overlapping subnet range (172.18.x.x)
These subnets may conflict with customer's Management network. We need a way override the default base IP Range used by Docker engine in the Gateway.

Resolution

VMware Engineering is working on this issue to provide further updates. 

For now, workaround can be implemented to resolve the issue 

Workaround:

1. Deploy the GW. If the GW is already deployed and registered to Cloud, it needs to be re-deployed.

2. SSH to GW 

3. Edit "/etc/docker/daemon.json". Specify the IP Range that does not conflict with customer's mgmt network as below:
   bip: Base IP Address range used by default bridge network
   default-address-pools: IP Range that will be used for any new bridge network that gets created. Here 172.21.0.0/16 denotes the full range of IPs (total 65534), and /24 means the the netmask that will be used for each created network 255.255.255.0). Ref: https://forums.docker.com/t/docker-default-address-pool-customization-question/112969

   Note: "If the conflicting network is only "172.17.0.0/16", then you can skip adding entry for "default-address-pools"

   [ ~ ]# more /etc/docker/daemon.json {  "bip": "172.20.0.1/16",  "default-address-pools": [     {"base":"172.21.0.0/16","size":24}   ] }

4. Reboot the GW

5. SSH back to GW after few seconds, and check if "docker ps" is not showing any error

   [ ~ ]# docker ps CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

6. Proceed with GW registration to Cloud

7. Once the GW Registration is complete, inspect the docker networks. Make sure that default bridge is using the subnet specified in "bip", and aap-custom-net is using subnet specified in "default-address-pools"

   [ ~ ]# docker network ls NETWORK ID     NAME             DRIVER    SCOPE 31e3456f7bc5   aap-custom-net   bridge    local 52c212fa103f   bridge           bridge    local 46bf152075c1   host             host      local 73a85c4189cc   none             null      local

   [ ~ ]# docker inspect bridge [     {         "Name": "bridge",         "Id": "52c212fa103f5390fe476e7153bc9c386a2d1e39ba12f17201fe24f452756e1f",         "Created": "2022-12-21T08:48:47.200005317Z",         "Scope": "local",         "Driver": "bridge",         "EnableIPv6": false,         "IPAM": {             "Driver": "default",             "Options": null,             "Config": [                 {                     "Subnet": "172.20.0.0/16",                     "Gateway": "172.20.0.1"                 }             ]         }, .....

   [ ~ ]# docker inspect aap-custom-net [     {         "Name": "aap-custom-net",         "Id": "31e3456f7bc524f4777dab841b75d988a98a1674f8076defe19a878ef2560214",         "Created": "2022-12-21T08:48:48.101976947Z",         "Scope": "local",         "Driver": "bridge",         "EnableIPv6": false,         "IPAM": {             "Driver": "default",             "Options": {},             "Config": [                 {                     "Subnet": "172.21.0.0/24",                     "Gateway": "172.21.0.1"                 }             ]         },         "Internal": false,         "Attachable": false,         "Ingress": false,         "ConfigFrom": {             "Network": ""         },         "ConfigOnly": false,         "Containers": {             "2c1fa07527b759f57588aea59f2fc3ab1a29519b340ba215b8db991ecfa909d4": {                 "Name": "ssh-rts-connection-agent-cedf14d5a4fe",                 "EndpointID": "243fd08e4bf73c21efc4a4c2b90c29c469d484f78915f840c96219e47c817a2b",                 "MacAddress": "##:##:##:##:##:08",                 "IPv4Address": "172.21.0.8/24",                 "IPv6Address": ""             }, .....

Additional Information

Impact/Risks:

If the default bridge network used by VMWare Cloud Gateway conflicts with customer's management network, VMWare Cloud Gateway UI will fail to load on browser. User will not be able to register GW and VC and subscribe to vSphere+

Impacted service is "vSphere+"