BOSH DNS Network Error Warnings After Diego Cell Scale Down

Products

• VMware Tanzu Application Service

Issue/Introduction

After scaling down Diego Cells, BOSH DNS HealthChecker WARN logs are observed across multiple deployments reporting no route to host errors for IPs belonging to deleted VMs.

Example log entries:

bosh-dns rs2 - [instance@47450 director="" deployment="p-rabbitmq-44af9267ca7aa3cc9175" group="rabbitmq-server" az="AZ-03" id="e798ed12-63a7-4c98-aec9-b8629366d44c"] [HealthChecker] 2026-02-23T16:29:02.323057535Z WARN - network error connecting to 10.##.###.###:8853: Performing GET request: Get "https://10.##.###.###:8853/health": dial tcp 10.##.###.###:8853: connect: no route to host

bosh-dns rs2 - [instance@47450 director="" deployment="p-isolation-segment-59622e940aee59422976" group="isolated_diego_cell" az="AZ-01" id="87d2bab9-ba82-4ea3-97cb-5522715d12a1"] [HealthChecker] 2026-02-23T17:16:29.266463377Z WARN - network error connecting to 10.##.###.###: Performing GET request: Get "https://10.##.###.###:8853/health": dial tcp 10.##.###.###:8853: connect: no route to host

bosh-dns rs2 - [instance@47450 director="" deployment="cf-6d3a8d41d1b395f8ef1b" group="diego_cell" az="AZ-01" id="a313c6ac-ac45-499f-a581-a78a38c7a226"] [HealthChecker] 2026-02-23T17:16:30.191834936Z WARN - network error connecting to10.##.###.###.158: Performing GET request: Get "https://10.##.###.###:8853/health": dial tcp 10.##.###.###:8853: connect: no route to host

The warnings can appear across multiple unrelated deployments such as p-rabbitmq, p-isolation-segment, and the cf deployment simultaneously, all referencing the same deleted cell IPs.

Environment

TAS v6.0.23+LTS-T

Cause

BOSH DNS is co-located on every VM and maintains a local health state table of all known VM IPs. It continuously polls each known IP on port 8853 to verify health. When Diego Cells are deleted as part of a scale down, the VMs and their IPs are removed from the IaaS immediately, but BOSH DNS may fail to clear the IPs. As a result, it continues to attempt a healthcheck to VMs that no longer exist, generating repeated no route to host WARN entries in the BOSH DNS logs.

The warnings are cosmetic in nature and do not impact running applications, but they persist until the stale health state is cleared.

Resolution

Step 1: Confirm the IPs no longer exist

• Verify via bosh vitals or bosh vms that the IPs referenced in the WARN logs do not correspond to any live VM:

  bosh -d <deployment-name> vms

• If the IPs are absent from all deployment VM lists, proceed to Step 2

Step 2: Restart BOSH DNS on the affected VMs

• bosh -d cf-<id> ssh diego_cell -c "sudo monit restart bosh-dns"