TX packet drops reported by vROPs for virtual machine on NSX-T segment
search cancel

TX packet drops reported by vROPs for virtual machine on NSX-T segment

book

Article ID: 322462

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:

  • Multiple VM reporting high TX drops in vROP on NSX-T segments.
  • No impact reported due to the packet drops
  • Viewing VM port stats using vsish, we see drops:
cat /net/portsets/DvsPortset-1/ports/67109164/inputStats
DVFILTER_VNIC_IN_GUEST <vmware-sfw:0x4311fee63512>
        pktsStarted:53880402
        pktsPassed:50530971
        pktsDropped:3349431
        pktsFiltered:0
        pktsQueued:0
        pktsFaulted:0
        pktsInjected:0
        pktErrors:0  
cat /net/portsets/DvsPortset-0/ports/67109164/vmxnet3/txSummary
stats of a vmxnet3 vNIC tx queue {
   generation:11223
   pkts tx ok:52789244
   bytes tx ok:9112557440
   TSO pkts tx ok:571801
   TSO bytes tx ok:2392954565
   unicast pkts tx ok:52789237
   unicast bytes tx ok:9112557146
   multicast pkts tx ok:0
   multicast bytes tx ok:0
   broadcast pkts tx ok:7
   broadcast bytes tx ok:294
   pkts tx failure:0
   pkts discarded:0
   error when copying hdrs:0
   tso header errors:0
   pkt allocation failures:0
   # of times a tx queue is stopped:0
   failed to map some guest buffers:0
cat /net/portsets/DvsPortset-0/ports/67109164/stats
packet stats {
pktsTx:1170369462 
pktsTxMulticast:0
pktsTxBroadcast:26
pktsRx:1223815657
pktsRxMulticast:0
pktsRxBroadcast:1163405
droppedTx:161590057
droppedRx:82201
  • Checking the DFW filter of the VM using summarize-dvfilter and vsipioctl
#summarize-dvfilter
world 72383945 vmm0:vx-4941097172 vcUuid:'50 0b 64 09 bc ## ## ##-## ## ## ## ## ## 80 dc'
port 67109164 vx-4941097172.eth0
vNic slot 2 name: nic-72383945-eth0-vmware-sfw.2
agentName: vmware-sfw
state: IOChain Attached
vmState: Attached
failurePolicy: failClosed
serviceVMID: 2
filter source: Dynamic Filter Creation
moduleName: nsxt-vsip-19761813
 
#vsipioctl getfilters -s
Filter Name              : nic-72383945-eth0-vmware-sfw.2
VM UUID                  : 50 0b 64 09 bc ## ## ##-## ## ## ## ## ## 80 dc
VNIC Index               : 0
VNIC UUID                : 500b6409-####-####-####-##########dc.000
VIF ID                   : c53ca8d1-####-####-####-##########c83
LSP ID                   : c5d61074-####-####-####-##########1b
Service Profile          : --NOT SET--
Filter Rule Config       : configured
Filter Hash              : 20132
Basic Stats:                From Switch  To Switch
   Total rx from dvfilter : 1199764347   1254357160
   Total packets          : 1199715455   1171952477
   Total time (us)        : 2020322128   12736318215
   Rate (us/pkt)          : 1            10
   Rate (pkt/s)           : 593823       92016
 
#vsipioctl getfilterstat -f nic-72383945-eth0-vmware-sfw.2
PACKETS                       IN                OUT
-------                       --                ---
v4 pass:                 1198259988         1171938619
v4 drop:                        336              18400
v4 reject:                    17230           82386448

v6 pass:                          0                  0
v6 drop:                          0                  0

BYTES                         IN                OUT
-----                         --                ---
v4 pass:               208158739203       200721582402
v4 drop:                      19520             956800
v4 reject:                   641217         4943255924

v6 pass:                          0                  0
v6 drop:                          0                  0

DROP REASON
-----------
src-limit:            211
strict no syn:        18736
3wh error:            7

FILTER INFO
-----------
sessions:       2350916
flags:          0xe46
states:         615
rules:          321
table count:    174
filter version: 1100
ruleset gen:    1542067
hash:           20132
last purge:     6458993
fprn alloc err: 0



Environment

VMware NSX-T Data Center

Cause

From above we can see a few packets are dropped due to not finding a SYN packet at the start of a new TCP flow, but this is not the issue here.
In the getfilterstat results for the filter, we can see a lot of transmitted packets are getting rejected:
PACKETS                   IN                 OUT
-------                   --                 ---
v4 pass:                  1198259988         1171938619
v4 drop:                         336              18400
v4 reject:                     17230           82386448

If we check the rules, we see there is one rule with a similar amount of out packets getting rejected:
#vsipioctl getrules -f nic-72383945-eth0-vmware-sfw.2 -s
rule     2 at 35, 396409 evals, 82402926 hits, 235 sessions, in 17230 out 82386444 pkts, in 641217 out 4943255740 bytes

The issue here is that the transmitted packet did not match any flows and eventually hit the default reject rule, which caused the counter to increment.

Resolution

This is expected behaviour, as the traffic did not match any rule and the default rule was reject.
To help narrow the issue down, you can enable logging on the default rule, then check the firewall logs on the ESXi host for the VM reporting packet drops:
/var/run/log/dfwpktlogs.log
Check the traffic flow for rejected rules and ensure they are correctly rejected and review traffic flow from the VM.

Workaround:
Make sure all traffic flows are matched with a rule.