Diego Cell VMs Have Duplicate DHCP Default Routes When Deployed with Multiple NICs on AWS (TAS on AWS / bosh-agent / systemd-networkd)
search cancel

Diego Cell VMs Have Duplicate DHCP Default Routes When Deployed with Multiple NICs on AWS (TAS on AWS / bosh-agent / systemd-networkd)

book

Article ID: 447057

calendar_today

Updated On:

Products

VMware Tanzu Application Service VMware Tanzu Platform VMware Tanzu Platform - Cloud Foundry VMware Tanzu Platform Core

Issue/Introduction

When Elastic Application Runtime (EAR/TAS) is deployed on AWS with multiple network interface cards (NICs) configured on the diego_cell instance group — using the BOSH multi-NIC feature available from AWS CPI v107.0.0+ — both NICs receive a DHCP-advertised default route (0.0.0.0/0) from AWS, even when the secondary NIC's subnet has no internet or NAT gateway route in its AWS route table.

This results in two competing default routes in the kernel routing table of each Diego Cell VM, causing non-deterministic outbound traffic behavior. Depending on which default route the kernel selects, application traffic, container-to-container networking, GoRouter communication, and BOSH agent heartbeats may egress via the wrong interface.

Running ip route show inside a Diego Cell shows two default gateway entries, one per NIC, both installed via DHCP:

default via x.x.x.x dev eth1 proto dhcp src x.x.x.x metric 1024
default via x.x.x.x dev eth0 proto dhcp src x.x.x.x metric 1024

Running route -n shows the same duplication under the Kernel IP routing table:

Destination     Gateway       Genmask         Flags Metric Ref  Use Iface
  0.0.0.0         x.x.x.x    0.0.0.0         UG    1024   0      0 eth1
  0.0.0.0         x.x.x.x    0.0.0.0         UG    1024   0      0 eth0

Environment

  • Elastic Application Runtime for VMware Tanzu Platform 10.4.x
  • Ops Manager 3.3.x
  • Stemcell: bosh-aws-xen-hvm-ubuntu-jammy-go_agent (Ubuntu 22.04, systemd-networkd)
  • AWS CPI with multi-NIC (additional_networks) enabled

Cause

This is a bosh-agent deficiency in the systemd-networkd network renderer.

When bosh-agent configures network interfaces on Ubuntu Jammy (which uses systemd-networkd ) it generates a .network unit file for every NIC. However, it generates identical files for all NICs regardless of whether a given network has been designated as the default gateway in the BOSH manifest.

For systemd-networkd, the [DHCP] key UseRoutes=no instructs the OS to accept the DHCP lease (IP address, MTU, DNS) but discard the DHCP-advertised default route. Without this key, systemd-networkd installs any default gateway it receives from the DHCP server — which is exactly what happens on AWS.

 

Resolution

A fix is required in cloudfoundry/bosh-agent to emit UseRoutes=no for non-default NICs when rendering systemd-networkd .network unit files.  This will be fixed in the upcoming versions of the stemcells.

Until a patched stemcell is available, we can use the below workaround:

SSH into each affected Diego Cell and run:

# 1. Patch the secondary NIC config
sudo sed -i '/\[DHCP\]/a UseRoutes=no' /etc/systemd/network/10_eth1.network

# 2. Verify the patch was applied
cat /etc/systemd/network/10_eth1.network

# 3. Reload systemd-networkd
#    NOTE: SSH session will drop momentarily. Reconnect after ~10 seconds.
sudo systemctl restart systemd-networkd

# 4. Verify — should now show only one default route
ip route show

You should only see one default route and that would be for eth0

 

Note: This workaround is ephemeral and wont persists upon a VM recreation.

Powered by Wolken Software