TX packet drops reported by Aria Operations for virtual machine on NSX segment
search cancel

TX packet drops reported by Aria Operations for virtual machine on NSX segment

book

Article ID: 322462

calendar_today

Updated On:

Products

VMware NSX VMware vDefend Firewall VMware vSphere ESXi

Issue/Introduction

  • Multiple VM reporting high TX drops in vROP on NSX segments.
  • No impact reported due to the packet drops
  • Viewing VM port stats using vsish, drops are seen at vmware-sfw in the I/O channel:
cat /net/portsets/DvsPortset-1/ports/67109164/inputStats
DVFILTER_VNIC_IN_GUEST <vmware-sfw:0x4311fee63512>
        pktsStarted:53880402
        pktsPassed:50530971
        pktsDropped:3349431
        pktsFiltered:0
        pktsQueued:0
        pktsFaulted:0
        pktsInjected:0
        pktErrors:0   
cat /net/portsets/DvsPortset-0/ports/67109164/vmxnet3/txSummary
stats of a vmxnet3 vNIC tx queue {
   generation:11223
   pkts tx ok:52789244
   bytes tx ok:9112557440
   TSO pkts tx ok:571801
   TSO bytes tx ok:2392954565
   unicast pkts tx ok:52789237
   unicast bytes tx ok:9112557146
   multicast pkts tx ok:0
   multicast bytes tx ok:0
   broadcast pkts tx ok:7
   broadcast bytes tx ok:294
   pkts tx failure:0
   pkts discarded:0
   error when copying hdrs:0
   tso header errors:0
   pkt allocation failures:0
   # of times a tx queue is stopped:0
   failed to map some guest buffers:0
cat /net/portsets/DvsPortset-0/ports/67109164/stats
packet stats {
pktsTx:1170369462 
pktsTxMulticast:0
pktsTxBroadcast:26
pktsRx:1223815657
pktsRxMulticast:0
pktsRxBroadcast:1163405
droppedTx:161590057  <---- Tx packets dropped 
droppedRx:82201
  • Checking the DFW filter of the VM using summarize-dvfilter and vsipioctl
#summarize-dvfilter
world 72383945 vmm0:vx-4941097172 vcUuid:'50 0b 64 09 bc ## ## ##-## ## ## ## ## ## 80 dc'
port 67109164 vx-4941097172.eth0
vNic slot 2 name: nic-72383945-eth0-vmware-sfw.2  <-- This is the DFW filter name
agentName: vmware-sfw
state: IOChain Attached
vmState: Attached
failurePolicy: failClosed
serviceVMID: 2
filter source: Dynamic Filter Creation
moduleName: nsxt-vsip-19761813
 
#vsipioctl getfilters -s
Filter Name              : nic-72383945-eth0-vmware-sfw.2
VM UUID                  : 50 0b 64 09 bc ## ## ##-## ## ## ## ## ## 80 dc
VNIC Index               : 0
VNIC UUID                : 500b6409-####-####-####-##########dc.000
VIF ID                   : c53ca8d1-####-####-####-##########c83
LSP ID                   : c5d61074-####-####-####-##########1b
Service Profile          : --NOT SET--
Filter Rule Config       : configured
Filter Hash              : 20132
Basic Stats:                From Switch  To Switch
   Total rx from dvfilter : 1199764347   1254357160
   Total packets          : 1199715455   1171952477
   Total time (us)        : 2020322128   12736318215
   Rate (us/pkt)          : 1            10
   Rate (pkt/s)           : 593823       92016
 
#vsipioctl getfilterstat -f nic-72383945-eth0-vmware-sfw.2
PACKETS                       IN                OUT   
-------                       --                ---
v4 pass:                 1198259988         1171938619
v4 drop:                        336              18400  <--- IPv4 packets that are dropped by rules with action set to DROP
v4 reject:                    17230           82386448  <--- IPv4 packets that are dropped by rules with action set to REJECT, or with rules using context-profiles

v6 pass:                          0                  0 
v6 drop:                          0                  0 <--- IPv6 packets that are dropped by a rules with action set to DROP 

BYTES                         IN                OUT -----                         --                --- v4 pass:               208158739203       200721582402 v4 drop:                      19520             956800 v4 reject:                   641217         4943255924 v6 pass:                          0                  0 v6 drop:                          0                  0 DROP REASON ----------- src-limit:            211 strict no syn:        18736 3wh error:            7
match drop rule rx packets: 1218580 <---- NSX 4.x and above have these two counters
match drop rule tx packets: 45985327 <---- NSX 4.x and above have these two counters

FILTER INFO ----------- sessions:       2350916 flags:          0xe46 states:         615 rules:          321 table count:    174 filter version: 1100 ruleset gen:    1542067 hash:           20132 last purge:     6458993 fprn alloc err: 0



Environment

VMware NSX-T Data Center
VMware NSX

Cause

  • From above its observed that few packets are dropped due to not finding a SYN packet at the start of a new TCP flow ( the "strict no syn counter" ), but is not the main reason for drops.
  • In the getfilterstat results for the filter, most of the transmitted packets are being dropped by Distributed Firewall Rules.
PACKETS                   IN                 OUT
-------                   --                 ---
v4 pass:                  1198259988         1171938619
v4 drop:                         336              18400  <--- IPv4 packets that are dropped by rules with action set to DROP
v4 reject:                     17230           82386448  <--- IPv4 packets that are dropped by rules with action set to REJECT, or with rules using context-profiles
  • In this particular issue, one rule shows a similar amount of out packets being dropped by the default rule, which has an action of REJECT.
#vsipioctl getrules -f nic-72383945-eth0-vmware-sfw.2 -s
rule     2 at 35, 396409 evals, 82402926 hits, 235 sessions, in 17230 out 82386444 pkts, in 641217 out 4943255740 bytes
  • DFW rules have three options for the Action  
    • ACCEPT - The DFW rule criteria is met and the action is set to ACCEPT, so the network flow is allowed
    • DROP -    The  DFW rule criteria is met and the action is set to DROP, so the network flow is silently dropped.  i.e. A reject packet is NOT sent back to the source IP
    • REJECT - The DFW rule Criteria is met and the action is set to REJECT, so a reject packet is sent back to the source IP.

         NOTE:  A DFW rule with action set to DROP and uses a context profile will always be shows as a REJECT in the getfilterstat output due to how the flow is processed.

This is the DFW working as expected.   The flow didn't match any other rules and, in this case, the default rule was set to action REJECT.

Resolution

This is expected behavior, as the traffic did not match any rule and the default rule action was REJECT.

How to identify which rules are being hit

  • For a quick test, you can add the VM to the DFW exclusion list to see if the alerts stop or are reduced significantly.

Additional Information

Alert "VM Network has dropped transmit packets" is triggering after applying Hot Fix 5 patch

High Packet Loss Observed in Aria Ops for Networks and vSphere Advanced Metrics when NSX IDS/IPS or L7 DFW is Enabled