Large number of "dropped_packets" comes from the Service Insertion Data Interface.

Products

• VMware NSX

Issue/Introduction

• When retrieving interface statistics using the GET API /api/v1/logical-router-ports/UUID/statistics/summary, a significant number of dropped packets are observed on the "SI_ServiceLinkPort" interface.
• When the interface stats are queried against Active and Standby Edge Nodes using “get logical-router interface stats”, we notice that the drops that we see from the above GET API, are recorded on the Standby Edge Nodes SI Interface whereas the Active Edge Node doesn’t indicate any drops.
• As a result, the forwarded statistics from the Manager to Aria Logs trigger an alert for packet drops on the SI interface, without distinguishing that these drops are actually originating from the Standby Edge Node.

Environment

VMware NSX.

Cause

• The packets that are recorded as “Dropped packets” on the Standby Edge Nodes SI Interface are SI BFD packets, these packets are Broadcast BFD control packets and will be exchanged between SI Interface peers. Since the T1 (Standby) interface is down, BFD packets sent from the standby will be dropped, which is expected behavior and does not indicate a problem.
• The GET API for interface statistics provides a cumulative output from both Edge Nodes, which is what gets forwarded to Aria Logs as well. There is no mechanism to filter statistics only from the Active Edge Node and forward it to Aria, so the data collected is cumulative and will be forwarded to syslog servers as a whole.

Resolution

The current behavior of the system is that dropped packets will inherently exist in an SI setup and will be visible on the Standby Edge Node's SI Interface.

Additionally, the NSX Manager aggregates statistics from both Active and Standby Edge Nodes, displaying a cumulative output from the GET API call. However, there is currently no mechanism to filter statistics based on Edge High Availability parameters and display statistics in the GET API for only the Active Node.

As a result, the alerts generated by Aria Logs are false positives. To confirm this, one can perform the following validation

Query the Active Node for on-demand statistics using the following API to verify whether the dropped packets are indeed originating from the Active Node. The same can be queried against the Standby Node to look at the dropped packets on the SI Interface:

GET /api/v1/logical-router-ports/<UUID>/statistics?transport_node_id=<TN_UUID>

To validate BFD packets, the following commands can be run on the Edge Nodes:

• get service-insertion >> to view the BFD status
• get bfd-session local-ip <IP address> remote-ip <IP address> >> to view BFD timers and other stats

Additionally, capturing packets on the Standby Edge's SI interface and verifying the SI BFD packets exchanged between SI Interface peers confirms that these packets are being blocked on the interface that is down and reported as dropped.