[VMC] VMs silently dropping packets while IDP/IDS is enabled

Products

• VMware Cloud on AWS

Issue/Introduction

To detail a common issue seen in VMC SDDCs when the IDPS feature is enabled.

Symptoms:

• Customer reports that multiple VMs in the SDDC will intermittently have slow/no network connectivity. This will be seen impacting multiple VM types, covering multiple Guest OS types. The impacted VMs from once incident may not be impacted during repeat incidents.
• The customer has IDS/IPS (Combined IDPS) NSX-T Advanced Firewall feature enabled for the Cluster(s) where the customer is facing VM impact.
• When viewing the Aria Operations for Applications graphs (formerly Wavefront), the impacted VMs will be seen dropping 50-100% of their packets during the impact timeframe. Correspondingly, the ESXi hosts will not report any packet losses during the period.

Cause

The distributed IDPS engine on individual ESXi hosts have a fixed capacity. Traffic exceeding this will be silently dropped.

Resolution

Immediate relief can be obtained by disabling the IDPS engine for the impacted Cluster.

Workaround:

The PDF attached to this KB article contains some fine-tuning recommendations and best practices to follow when implementing IDPS in VMC. Work with your Account Team and Professional Services for assistance in configuring the environment per best practices.

Additional Information

Impact/Risks:
VMs will silently have dropped packets.