How To Change The VIC Appliance Internal Bridge Network Configuration

Products

• VMware vSphere ESXi

Issue/Introduction

Internal networking inside the VIC Appliance, the docker0 interface for the Appliance Containers, uses range 172.17.0.0/16

Symptoms:

• VIC Appliance is not functioning as designed
• You already have physical network range using 172.17.0.0/16

Environment

VMware vSphere Integrated Containers 1.5.x
VMware vSphere Integrated Containers 1.4.x

Cause

The default internal networking for Docker Engine

Resolution

To modify the default internal docker engine networks a system modification is necessary. To make the system changes please follow the steps outlined below.

In order to apply the change, modify the 3 .yml files in the VIC Appliance with the correct range :

• /etc/vmware/harbor/docker-compose.yml
• /etc/vmware/harbor/docker-compose.clair.yml
• /etc/vmware/harbor/docker-compose.notary.yml

Below is an example of a desired network range that would be used in an environment. Based on these network range examples the .yml files will need to be modified to allow for your requirements.
VIC 1.4.x has 5 Harbor networks, where VIC 1.5.x only has 4 (VIC 1.5.x does not use harbor_notary-mdb any longer)

• 192.168.8.0/24 bridge
• 192.168.9.0/24 harbor_harbor
• 192.168.10.0/24 harbor_harbor-clair
• 192.168.11.0/24 harbor_harbor-notary
• 192.168.12.0/24 harbor_notary-mdb
• 192.168.13.0/24 harbor_notary-sig

Make sure that SSH access to the appliance is enabled. To enable SSH access to the appliance, see Reconfigure the vSphere Integrated Containers Appliance.

Steps to perform :
1. SSH into the VIC Appliance as root
2. Modify the "network" section of the following files as indicated :

a. Edit the /etc/vmware/harbor/docker-compose.yml :

vim /etc/vmware/harbor/docker-compose.yml

b. Modify the networks section at the start of the /etc/vmware/harbor/docker-compose.yml file as following :

networks:
  harbor:
    external: false
    ipam:
      driver: default
      config:
      - subnet: 192.168.9.0/24
        gateway: 192.168.9.1
services:
...

c. Edit the /etc/vmware/harbor/docker-compose.clair.yml file :

vim /etc/vmware/harbor/docker-compose.clair.yml

d. Modify the networks section at the start of the /etc/vmware/harbor/docker-compose.clair.yml file as following :

networks:
  harbor-clair:
    external: false
    ipam:
      driver: default
      config:
      - subnet: 192.168.10.0/24
        gateway: 192.168.10.1
services:
...

e. Edit the /etc/vmware/harbor/docker-compose.notary.yml file :

vim /etc/vmware/harbor/docker-compose.notary.yml

f. Modify the networks section at the start of the /etc/vmware/harbor/docker-compose.notary.yml file as following :
- VIC 1.4.x :

networks:
  harbor-notary:
    external: false
    ipam:
      driver: default
      config:
      - subnet: 192.168.11.0/24
        gateway: 192.168.11.1
  notary-mdb:
    external: false
    ipam:
      driver: default
      config:
      - subnet: 192.168.12.0/24
        gateway: 192.168.12.1
  notary-sig:
    external: false
    ipam:
      driver: default
      config:
      - subnet: 192.168.13.0/24
        gateway: 192.168.13.1
services:
...

- VIC 1.5.x :

networks:
  harbor-notary:
    external: false
    ipam:
      driver: default
      config:
      - subnet: 192.168.11.0/24
        gateway: 192.168.11.1
  notary-sig:
    external: false
    ipam:
      driver: default
      config:
      - subnet: 192.168.13.0/24
        gateway: 192.168.13.1
services:
...

3. Stop containers and removes containers and networks by executing this command :

/usr/local/bin/docker-compose -f /etc/vmware/harbor/docker-compose.yml \
                              -f /etc/vmware/harbor/docker-compose.notary.yml \
                              -f /etc/vmware/harbor/docker-compose.clair.yml down

4. Create containers and networks and starts containers by executing this command :

/usr/local/bin/docker-compose -f /etc/vmware/harbor/docker-compose.yml \
                              -f /etc/vmware/harbor/docker-compose.notary.yml \
                              -f /etc/vmware/harbor/docker-compose.clair.yml up &

5. Once this is complete, allow 10 minutes for all the Harbor components to start up.
6. Create the following file :

vim /etc/docker/daemon.json

7. Put the following content into it :

{
  "bip": "192.168.8.1/24",
  "fixed-cidr": "192.168.8.0/24"
}

8. Reboot the VIC Appliance
9. Log back in using SSH to the VIC Appliance and check the network, using this command :

docker network inspect harbor_harbor

You should see the newly define range persisting reboots.

Please note : If you perform a VIC Upgrade, this will not persist, and will have to be re-applied!

Additional Information

Impact/Risks:
There will be a system outage when making network changes to the VIC appliance. This will affect all running containers in the environment.