DFW Rule Statistics Not Updating on NSX-T Manager

Products

VMware vDefend Firewall

Issue/Introduction

This article explains an issue where NSX-T Manager shows zero statistics for DFW rules, even though the rules are being evaluated and hit, as evidenced by host logs and CLI outputs. This discrepancy occurs due to miscommunication between the NSX Manager and ESXi hosts or NSX Manager and Edge nodes.

Relevant Log Entries:

From NSX Manager Logs:

2025-01-21T20:43:08.689Z Er(179) nsx-exporter[2100545]: NSX 2100545 - [nsx@6876 comp="nsx-esx" subcomp="agg-service" tid="2100677" level="ERROR" errorCode="MPA10014"] [GetNextResponseInfo] Incoming pull stat request is invalid

From ESXi Host Logs:

nsx-exporter: NSX 8597797 - [nsx@6876 comp="nsx-esx" subcomp="agg-service" level="ERROR" errorCode="MPA10014"] [SummationStatsRequestHandler][IDPSRuleStats]: Bad summation stats request: -1

From NSX Edge Logs (Syslog):

2025-01-29T14:14:42.419Z xxxxxxxxxx NSX 1876 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="2161" level="INFO"] [GetNextResponseInfo] Incoming pull stat request with seq:159705, ack:142484 
2025-01-29T14:14:42.419Z xxxxxxxxxx NSX 1876 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="2161" level="INFO"] [GetNextResponseInfo] Current internal _lastSent: 125, _lastAck: 124 
2025-01-29T14:14:42.419Z xxxxxxxxxx NSX 1876 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="2161" level="ERROR" errorCode="MPA10014"] [GetNextResponseInfo] Incoming pull stat request is invalid 
2025-01-29T14:14:42.419Z xxxxxxxx NSX 1876 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="2161" level="ERROR" errorCode="MPA10014"] [SummationStatsRequestHandler][AggSvc-Routing]: Bad summation stats request: -1 
2025-01-29T14:14:42.419Z xxxxxxxxxx NSX 1876 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="2161" level="WARNING"] [DequeueRxCallbackMsg] ProcessRxCallbackMsg failed for app [AggSvc-Routing]

Environment

VMware vDefend Firewall

Cause

The issue arises from a mismatch in how rule statistics are synchronized between the NSX Manager and the ESXi hosts. Specifically:

The Manager fails to update DFW rule statistics despite the rules being actively evaluated on hosts.
Errors in the nsx-exporter service on ESXi hosts prevent the statistics from being sent to the Manager.
On NSX Edge nodes, similar alarms may occur due to service-related disruptions in node-stats.

Resolution

API's to validate the Information:

Use the API to verify the rule statistics on the affected ESXi hosts:

curl -k -u https://<host-IP>/api/v1/firewall/sections/<section-id>/rules/<rule-id>/stats

Use the following API to verify the connectivity of services:

curl -k -u -X POST "https://<NSX-Edge-IP>/api/v1/directory/ldap-server?action=CONNECTIVITY"

Use the following API to Check policy-level statistics:

curl -k -u https://<manager-IP>/policy/api/v1/infra/domains/default/security-policies/<policy-id>/statistics

Use the following API to Check manager-level statistics:

curl -k -u https://<manager-IP>/api/v1/firewall/sections/<section-id>/rules/<rule-id>/stats

If this issue is observed, please open an SR with Broadcom Support so that we can determine a full RCA.

If the issue needs to be resolved immediately, please follow these workaround steps:

Steps to Resolve on ESXi Hosts:

Restart the nsx-exporter service on the affected ESXi hosts.

bash

/etc/init.d/nsx-exporter restart
Verify that the errors (e.g., "Bad summation stats request") are no longer present in the Host logs.

grep 'MPA10014" /var/run/nsx-syslog.log

Steps to Resolve on NSX Edge Nodes:

Restart the node-stats service on the affected edges.

restart service node-stats
Validate that the alarms are cleared on the Edge and that statistics are syncing properly.

grep 'MPA10014" /var/log/syslog