UnknownHostException in app logs corrected by restart of bosh-dns on Diego Cells

Products

• VMware Tanzu Platform

Issue/Introduction

Customer sees UnknownHostException in app logs.

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://availability-api.example.com/v2/inventoryapis/searchinventory": availability-api.example.com: Name or service not known; nested exception is java.net.UnknownHostException: availability-api.example.com: Name or service not known

Although there is no reason to believe this is a bug in bosh-dns, the customer observed that after identifying the Diego cells on which the application instances were running and restarting the bosh-dns service on those VMs, the error was eliminated.

Environment

TAS 4.0.21

Cause

Restarting bosh-dns may simply cause DNS lookups to hit a different DNS resolver. 

Resolution

Performing general networking troubleshooting should be the first step. Problems with the functionality or data available to DNS servers are the most common cause for a "name or service not known" error.

If no problems are found in infrastructure DNS, then another workaround is to restart the bosh-dns job on the affected diego cell(s).

1. Lookup Diego cells where apps are running.
   a. Run this command: cf app <app_name> --guid . It shows the app_guid.
   b. Run this command: cf curl /v2/apps/<app_guid>/stats or cf curl /v3/processes/<app_guid>/stats . It shows the host IP address and port of the Diego cell in which the app instance is running.
   c. Run the command bosh vms | grep <host>  to retrieve the diego_cell name where host is the IP address listed in step b.
2. bosh ssh to Diego cell(s)
3. sudo -i
4. monit restart bosh-dns

If you want to gather more information and possible report the issue to R&D, you can collect a packet capture:

sudo tcpdump -i any port 53 -w dns_capture.pcap