Load balancer error detected by tcpdump packet capture

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

A connection reset issue was observed in the logs for an application. The exact error message was "CAU-Service-Exception - Exception occured in Request Handler:Connection reset by peer". 

Increasing CPU and memory on the load balancer instances eliminated the errors.

Environment

Product Version: 2.7

Resolution

We collected tcpdump packet captures on the tcp_router VMs in order to determine where the RST packets were originating. 

To collect packets for a particular application, determine which port(s) the app uses:

cf app <appname>

We found the relevant ports in the route(s) URLs. Then we used a tcpdump command with these options:

nohup tcpdump -C 256 -W 4 -s 256 -w /tmp/mysql-traces/wsrep -i eth0 “tcp port 9046 and port 9047”

We stopped the tcpdump when the relevant error was detected. This can be done by tailing and grepping for strings in local logs, or by using a log aggregator and a search filter (in Splunk, for example).

In this case, the RST packets were coming from the load balancer.

Increasing resources (CPU, memory) on load balancer instances resolved the connect resets.