DHCP clients can not get an IP address through DHCP relay when IDS/IPS is enabled on the same Tier-1 gateway

Products

• VMware NSX

Issue/Introduction

• You configured a DHCP relay on a Tier-1 gateway.
• You also enabled IDS/IPS on the same Tier-1 gateway.
• DHCP clients under the Tier-1 gateway can not get an IP address through the DHCP relay on the Tier-1 gateway.
• DHCP packets from the clients are forwarded to the DHCP server, but DHCP packets from the DHCP server are not forwarded to the clients from the Tier-1 gateway.
• If you keep DHCP clients running for long time, they finally get an IP address, but it usually takes minutes.
• You find "type 44" in the Edge logs when DHCP packets pass the Tier-1 gateway.

  /var/log/syslog
  <Timestamp> <Hostname> NSX 5024 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewall" level="INFO"] [IDS]: ids_attr_cb : 0x<Hex value> adding attribute type 44 xtn id 0 xtn flag 1

• When you disable IDS/IPS on the Tier-1 gateway, the DHCP clients get an IP address successfully and immediately.

Environment

VMware NSX-T 3.2

Cause

IDS/IPS wrongly classifies DHCP packets as an unknown protocol.
"type 47" is correct.

<Timestamp> <Hostname> NSX 4571 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewall" level="INFO"] [IDS]: ids_attr_cb : 0x<Hex value> adding attribute type 47 xtn id 0 xtn flag 1

IDS/IPS buffers the content of such an unknown protocol until max_inspection_bytes_unknown reaches, and then packets bypass IDS/IPS.

Resolution

Resolved in NSX 4.1.

NSX 4.2 and later is recommended because it has a new engine of IDS/IPS on Edge.

As a workaround, you can configure the value of max_inspection_bytes_unknown on the Edge nodes.

To get the current value, invoke the command. The default value is 4096.

root@<Hostname>:~# edge-appctl -t /var/run/vmware/edge/dpd.ctl  ids/getengineparams
{"ids_engine_profile":{"payloadsz": 8192,"inspectionsz": 8192,"unknownsz": 4096,"slowmatch": 1,"payloadeval": 1,"bypassunknown": 1,"combinationlogic": 0,"recursivematch": 1,"idsbypassclear": 0}}

If reduced to 1, DHCP clients get an IP address immediately.

root@<Hostname>:~# edge-appctl -t /var/run/vmware/edge/dpd.ctl  ids/setengineparams unknownsz 1
{"result":"success"}

Note:
The workaround does not persist after reboot.