Rate Limiting vDefend Firewall Packet Logs
Article ID: 396587
Updated On:
Products
Issue/Introduction
vDefend Distributed Firewall rules are created by using the NSX UI or API. The rules are published from the NSX Manager to the ESXi hosts where the VMs reside.
For aiding in constructing the rules as well as for troubleshooting the rulesets, the user can enable the packet logging feature at the rule level. When logging is enabled, packet information is logged into the log file /var/log/dfwpktlogs.log on the ESXi host and optionally exported to the external syslog server.
When logging is enabled on frequently hit rules, it generates a lot of packet logs, which affects some of the internal communication between ESXi host processes. As a result, the ESXi host may lose connectivity to the NSX Manager and/or vCenter.
Some daemons on the ESXi host, like nsx-opsagent/nsx-proxy/nsx-cfgagent/nsx-netopa/nsx-exporter/nsx-vdpi/nsx-nestdb, may get affected. The following is a sample of the keep-alive failures seen between the ESXi host processes.
2024-02-16T16:05:52.650Z nsx-opsagent[2107207]: NSX 2107207 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsx-rpc" tid="2107362" level="ERROR" errorCode="RPC31"] RpcConnection[465 Connected on tcp://127.0.0.1:4554 0] Keepalive failed - haven't received response in time (last request was sent 60 seconds ago, response received - 239 seconds ago)
2024-02-16T16:06:02.046Z nsx-opsagent[2107207]: NSX 2107207 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsx-rpc" tid="2107362" level="ERROR" errorCode="RPC31"] RpcConnection[466 Connected to tcp://127.0.0.1:2480 0] Keepalive failed - haven't received response in time (last request was sent 59 seconds ago, response received - 239 seconds ago)
2024-02-16T16:09:50.199Z vdpi[2107501]: NSX 2107501 - [nsx@6876 comp="nsx-esx" subcomp="nsx-vdpi" s2comp="nsx-rpc" tid="2107536" level="ERROR" errorCode="RPC31"] RpcConnection[78 Negotiating to tcp://127.0.0.1:2480 0] Keepalive failed - haven't received response in time (last request was sent 59 seconds ago, response received - never)
2024-02-14T01:44:15.463Z nestdb-server[2106894]: NSX 2106894 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" s2comp="nsx-rpc" tid="2106926" level="ERROR" errorCode="RPC31"] RpcConnection[1 Connected on tcp://0.0.0.0:2480 0] Keepalive failed - haven't received response in time (last request was sent 59 seconds ago, response received - 299 seconds ago)
Environment
VMware NSX 3.x
VMware NSX 4.x
Cause
The ESXi syslog daemon has a single queue that processes the syslog-related messages from various daemons. When vDefend Firewall sends excessive logs to this daemon, this queue can get congested and will be unable to process the messages.
Resolution
For environments using VMware NSX 4.2.1 and prior
The user can disable the vDefend Firewall packet logging for the specific rules which generate more packet logging.
For environments using VMware NSX 4.2.2 and above
By default, vDefend Firewall packet logging is capped to 10,000 packet logs / second per host.
The following log gets printed every 30 minutes in the ESXi host's vmkernel.log. When the packet logs are dropped for exceeding the limit, the "Dropped" counter would increase.
VSIP DFW: Log request HWM during 1800 sec period = 32631 LPS. Rate limit = 10000 LPS. Logged = 17500. Dropped = 154686.
Feedback
