TX packet drops reported by vROPs for virtual machine on NSX-T segment
Article ID: 322462
Updated On:
Products
VMware NSX Networking
Issue/Introduction
Symptoms:
- Multiple VM reporting high TX drops in vROP on NSX-T segments.
- No impact reported due to the packet drops
- Viewing VM port stats using vsish, we see drops:
cat /net/portsets/DvsPortset-1/ports/67109164/inputStats
DVFILTER_VNIC_IN_GUEST <vmware-sfw:0x4311fee63512>
pktsStarted:53880402
pktsPassed:50530971
pktsDropped:3349431
pktsFiltered:0
pktsQueued:0
pktsFaulted:0
pktsInjected:0
pktErrors:0
cat /net/portsets/DvsPortset-0/ports/67109164/vmxnet3/txSummary
stats of a vmxnet3 vNIC tx queue {
generation:11223
pkts tx ok:52789244
bytes tx ok:9112557440
TSO pkts tx ok:571801
TSO bytes tx ok:2392954565
unicast pkts tx ok:52789237
unicast bytes tx ok:9112557146
multicast pkts tx ok:0
multicast bytes tx ok:0
broadcast pkts tx ok:7
broadcast bytes tx ok:294
pkts tx failure:0
pkts discarded:0
error when copying hdrs:0
tso header errors:0
pkt allocation failures:0
# of times a tx queue is stopped:0
failed to map some guest buffers:0
cat /net/portsets/DvsPortset-0/ports/67109164/stats
packet stats {
pktsTx:1170369462
pktsTxMulticast:0
pktsTxBroadcast:26
pktsRx:1223815657
pktsRxMulticast:0
pktsRxBroadcast:1163405
droppedTx:161590057
droppedRx:82201
- Checking the DFW filter of the VM using summarize-dvfilter and vsipioctl
#summarize-dvfilter world 72383945 vmm0:vx-4941097172 vcUuid:'50 0b 64 09 bc 0e 73 ba-61 b1 89 8c 9b a5 80 dc' port 67109164 vx-4941097172.eth0 vNic slot 2 name: nic-72383945-eth0-vmware-sfw.2 agentName: vmware-sfw state: IOChain Attached vmState: Attached failurePolicy: failClosed serviceVMID: 2 filter source: Dynamic Filter Creation moduleName: nsxt-vsip-19761813
#vsipioctl getfilters -s Filter Name : nic-72383945-eth0-vmware-sfw.2 VM UUID : 50 0b 64 09 bc 0e 73 ba-61 b1 89 8c 9b a5 80 dc VNIC Index : 0 VNIC UUID : 500b6409-bc0e-73ba-61b1-898c9ba580dc.000 VIF ID : c53ca8d1-d903-4f4a-9d4a-0245aa846c83 LSP ID : c5d61074-3ca1-42ef-9c4a-c5b0809ec81b Service Profile : --NOT SET-- Filter Rule Config : configured Filter Hash : 20132 Basic Stats: From Switch To Switch Total rx from dvfilter : 1199764347 1254357160 Total packets : 1199715455 1171952477 Total time (us) : 2020322128 12736318215 Rate (us/pkt) : 1 10 Rate (pkt/s) : 593823 92016
#vsipioctl getfilterstat -f nic-72383945-eth0-vmware-sfw.2 PACKETS IN OUT ------- -- --- v4 pass: 1198259988 1171938619 v4 drop: 336 18400 v4 reject: 17230 82386448 v6 pass: 0 0 v6 drop: 0 0 BYTES IN OUT ----- -- --- v4 pass: 208158739203 200721582402 v4 drop: 19520 956800 v4 reject: 641217 4943255924 v6 pass: 0 0 v6 drop: 0 0 DROP REASON ----------- src-limit: 211 strict no syn: 18736 3wh error: 7 FILTER INFO ----------- sessions: 2350916 flags: 0xe46 states: 615 rules: 321 table count: 174 filter version: 1100 ruleset gen: 1542067 hash: 20132 last purge: 6458993 fprn alloc err: 0
Environment
VMware NSX-T Data Center
Cause
From above we can see a few packets are dropped due to not finding a SYN packet at the start of a new TCP flow, but this is not the issue here.
In the getfilterstat results for the filter, we can see a lot of transmitted packets are getting rejected:
If we check the rules, we see there is one rule with a similar amount of out packets getting rejected:
The issue here is that the transmitted packet did not match any flows and eventually hit the default reject rule, which caused the counter to increment.
In the getfilterstat results for the filter, we can see a lot of transmitted packets are getting rejected:
PACKETS IN OUT ------- -- --- v4 pass: 1198259988 1171938619 v4 drop: 336 18400 v4 reject: 17230 82386448
If we check the rules, we see there is one rule with a similar amount of out packets getting rejected:
#vsipioctl getrules -f nic-72383945-eth0-vmware-sfw.2 -s rule 2 at 35, 396409 evals, 82402926 hits, 235 sessions, in 17230 out 82386444 pkts, in 641217 out 4943255740 bytes
The issue here is that the transmitted packet did not match any flows and eventually hit the default reject rule, which caused the counter to increment.
Resolution
This is expected behaviour, as the traffic did not match any rule and the default rule was reject.
To help narrow the issue down, you can enable logging on the default rule, then check the firewall logs on the ESXi host for the VM reporting packet drops:
Workaround:
Make sure all traffic flows are matched with a rule.
To help narrow the issue down, you can enable logging on the default rule, then check the firewall logs on the ESXi host for the VM reporting packet drops:
/var/run/log/dfwpktlogs.log
Check the traffic flow for rejected rules and ensure they are correctly rejected and review traffic flow from the VM.Workaround:
Make sure all traffic flows are matched with a rule.
Feedback
Yes
No
Powered by
