Using Wireshark with Pivotal Products

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Wireshark can be used to debug and troubleshoot a number of Pivotal products. This article provides a high-level overview of Wireshark usage and packet capturing.

Note: Concerns regarding specific products are addressed in other articles.

Environment

Resolution

Capturing packets with Wireshark

Wireshark includes capturing capabilities that work like other capture tools. If listening wide-open, also known as "promiscuous", a sniffer can capture any packets on the local network. Filter the captured packets between the client and the server of interest.

The following is an example capture filter:

host 172.31.0.100 and port 80

The capture filter will:

Capture all traffic to and from a Hypertext Transfer Protocol (HTTP) server by listening to the host and port.
Capture traffic from the host to port 80 on another server. You may get more traffic than expected due to web service call outs and yum repository checks.

Capturing packets with other tools

You may be more comfortable capturing traffic with tools that are already in place. You do not need to use Wireshark to capture packets, it can read output from most tools.

You can capture and filter packets using `tcpdump`. The `-w` option writes packets to a file which can be read by Wireshark.

tcpdump -n -s 0 -w tcpdump.out port 8080

Traffic can be filtered at the point of capture but it is better to overcapture and apply filters afterward, while viewing in Wireshark. Capture as much as possible and ensure that the capture file sizes are small enough to transfer and load into Wireshark.

Capture full packets

Wireshark captures full packet contents by default, allowing us to see the HTTP and AJP packet contents in addition to Transmission Control Protocol (TCP) headers.

Other tools may require specification in order to capture full packets. You can tell the tool to capture full packets using the `-s 0` option, however most new versions of `tcpdump` capture full packets default.

tcpdump: tcpdump -n -s 0 -w tcpdump.out

Wireshark UI basics

The Wireshark UI recalls prior actions or commands and executes them again. The following actions are remembered by Wireshark:

Note: these globals are listed in the preferences tab

Settings: columns in main view and checksum validation.
Expanded or contracted packet sections: when moving between packets in a trace, Wireshark attempts to move you into the same place
Protocol decoding preferences: Wireshark will remember how to decode traffic to odd ports once those preferences have been set.

Good things to customize:

Preferences: disable checksums on Internet Protocol (IP) and Transmission Control Protocol (TCP) packets. The checksums after often inaccurate because the Operating System (OS) handles the capture and NIC handles the sum.
Disable TCP transport resolution: it attempts to name the ports and will often become disruptive.
Columns displayed: navigate to preferences, UI, and to columns to customize the columns

Analyzing the trace

Note: View filters possess syntax which differ from capture filters. "=" is used for capture filters, "==" is used for view filters. In the view filter, invalid expressions will have a red background and valid expressions will have a green background.

Filter out IP & port if needed:
- ip.src/dest/addr == x.x.x.x, tcp.srcport/dstport/port == 8080
- If src or dst aren't specified, both ends of the connection are considered a match. This produces unexpected results when attempting to add filters such as: ip.addr != 1.2.3.4, one end will always match. You may require !(ip.addr == 1.2.3.4).
View as protocol such as HTTP or AJP13:
- Right-click the packet and select decode as...
- Choose the transport tab
- Decode TCP on ports HTTP and AJP13 or others
Working with SSL, TLS, and HTTPS
- Without decrypting traffic, the TCP headers and the TLS or SSL negotiation will be visible. The actual data is noise.
- Wireshark can decode encrypted traffic if you have the private server key, see the Wireshark wiki for more information.
Isolate a TCP stream, which represents one connection from start to close
- Filter tcp.stream eq 0
- The stream number is the stream index under TCP
- Alternatively, isolate a stream and add a filter by right clicking on either follow tcp stream or filter out this stream
TCP basics, more detail can be found on this wikipedia page
- Levels lower than TCP or IP can be ignored
- TCP is the control mechanism or envelope for the conversation
- Starting a connection: 3-way handshake SYN SYN-ACK ACK packets
- Data is observed from the Protocol Data Units (PDU) within the contents of the conversation
- Receiver acknowledges sent packets with ACK, which may refer to multiple packets listed under TCP information
- Close connection: up-to-4-way handshake FIN-ACK ACK FIN-ACK ACK, flags may be included in packets with PDU contents, especially with AJP.
Isolate data rather than controlling packets:
- (tcp.stream eq 0) and http
- (tcp.stream eq 0) and ajp13
- TCP segment of a reassembled PDU is one of a series of packets from a PDU that has been broken up. The final content packets are rolled up and displayed. You may want to look at NOT protocol, such as !http, to find errant packets such as control packets and incomplete PDUs.
Filters drill down into protocol attributes:
- http.response.code == 500
- http.cookie contains "JSESSIONID=xyz"
- You can also filter on contained protocols such as XML attributes in a SOAP message sent over HTTP
Saving out selected packets for separate analysis or escalation:
- Be aware that Wireshark may not be able to interpret the protocol properly with partial packets or if it is missing one side of a conversation.
- Mark desired packets by clicking on Edit -> Mark all displayed packets. Alternatively, you can mark the first and last by right clicking Mark packet
- To mark from first to last, click on File -> Save as -> Marked
Saving out packet contents:
- You can highlight the desired contents in the view of the packet by right clicking export selected packet bytes
- You can see protocol contents of a stream by clicking follow TCP stream

See the Wireshark wiki for additional examples and information.

Additional Information

Comparing packet traces: WireShark vs. browser header plugins or Fiddler:

Wireshark advantages:

Unobtrusive observing: no need to install or configure within browsers or applications
Low-level network examination and HTTP-level content: Wireshark can observe keepalive timeouts
Understands AJP, RMI, AMQP. Does not understand HTTP.

Wireshark disadvantages:

Fiddler can interpret HTTPs easily through the use of MSIE browser plugins and proxy. Wireshark requires an RSA key to interpret HTTPs.
In some instances, Fiddler can drill into content layers more effectively. Wireshark may choke on chunked GZIP traffic.
Wireshark has a large bug backlog, with bugs more apparent than others.