ESXi not responding due to high DFW log rate

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host shows as "not responding" state in vCenter.
• ESXi hostd service is non-responsive.

vmkernel log:

Al(177) vmkalert: cpuxx:xxxx)ALERT: hostd detected to be non-responsive
Al(177) vmkalert: cpuxx:xxxx)ALERT: hostd detected to be non-responsive

• DFW (nsx distributed firewall) log recorded a lot of logs, and mainly on one kind of flow. In following example, the records are with same source ip, destination ip and UDP port 514.

dfwpktlogs.log
No(13) FIREWALL-PKTLOG[14823352]: 7c636119 INET match REJECT xxx IN xxx UDP x.x.x.x/xxx->x.x.x.x/514 xxxDeny
No(13) FIREWALL-PKTLOG[14823352]: 7c636119 INET match REJECT xxx IN xxx UDP x.x.x.x/xxx->x.x.x.x/514 xxxDeny
No(13) FIREWALL-PKTLOG[14823352]: 7c636119 INET match REJECT xxx IN xxx UDP x.x.x.x/xxx->x.x.x.x/514 xxxDeny

vmkernel.log

vmkernel: cpuxx:xxxx)VSIP DFW: Log request HWM during 1800 sec period = 12083 LPS. Rate limit = 10000 LPS. Logged = 15886423. Dropped = 81622.

• ESXi hostd service performance degraded.

vmkernel.log

Al(177) vmkalert: cpuxx:xxxx)ALERT: hostd performance has degraded due to high system latency

Environment

VMware vSphere ESXi

Resolution

• Disable DFW logging.
• Stop the abnormal flow recorded in dfwpktlogs.log.