Considerations to run pktcap-uw under heavy network load

Products

• VMware vSphere ESXi

Issue/Introduction

The focus for this article is to provide advise and best practices when using the pktcap-uw tool when the ESXi hosts are heavily loaded.

You see the following messages on vmkernel.log (/var/log) and ESXi host disconnect from vCenter: 

• Packet dup failed for packet Out of memory
• Failed to linearize packet in txq
• Failed to alloc rx buff
• Failed to alloc mem for
• FastSlab VASpace exhausted on node
• Host vSAN membership reduced: Current membership uuid xxxxx-xxxx-xxxxx-xxxxx has 1 members
  
  

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

The pktcap-uw tool, needs to duplicate packets in memory, hold the packets and then send to user world, under high load this process can be costly to memory.

Resolution

• Avoid running pktcap-uw with multiple pipes (|), for example:

#pktcap-uw --uplink <vmnic> --capture UplinkSndKernel,UplinkRcvKernel --ng -o - | tcpdump-uw -enr - | grep <ip_address> | gzip -c - > /vmfs/volumes/<datastore>/<directory>/uplink-capture-<vmnix>.txt.gz & 

• Use a filter option, example:

# pktcap-uw --uplink <vmnic> --ip <ip_address> -o capturefile.pcap

• Use short snap (length of the packet to capture), example:

#pktcap-uw --uplink vmnicX --snap 200

• Avoid running more than 4 (four) pktcap-uw sessions simultaneously in the background.
  
  
• If you need to stop captures running in the background use the following command: #kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
  
  
• If you are running captures on an uplink avoid saving pktcap-uw output to a network datastore using that same uplink

Additional Information

Using the pktcap-uw tool in ESXi 5.5 and later

Improper termination of pktcap-uw causes Userworld exhaustion and ESXi host management agent failures