Alarm for IDPS message transmission failure

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Alarm for IDPS Message transmission failure
Event ID: distributed_ids_ips.message_transmission_failure

Alarm Description

• Purpose: Data Transmission failure between ESX host and messaging broker

Impact: Since there is a message transmission failure between the ESX host and the messaging broker, some IDPS messages and / or PCAPs may be lost.

• Check for connectivity issues between NSX Manager and the ESX host.

For NAPP

• Issue the NAPP health API https://< NSX MANAGER >/napp/api/v1/platform/monitor/feature/health on the NSX manager and check if the component by the name 'kafka' appears as healthy. If kafka is unhealthy, check output of the following commands on the NSX Manager.
  
  napp-k get pods -o wide | grep kafka


• If kafka is unhealthy, contact Broadcom support.
• If kafka is healthy, check for SSL handshake error between IDPS agent and kafka broker. Keywords to look for – 'SSL handshake failed' and 'Nsxi: Kafka broker(s) are down.' in /var/log/nsx-syslog. If there is an SSL handshake error, restart proton on the node where the service COMMON_AGENT_SERVICE is the leader. You can determine where this service is the leader by issuing the 'get cluster status verbose' command on the NSX Manager.
  
  
• Check if there is a network policy preventing communication between the transport node where the alarm is seen and the kafka broker.
  
  napp-k get svc | grep external 
  
  Check connectivity from affected host to all the kafka external IPs.
  
  nc -vz <messaging IP> 9092

FOR SSP

• check the output of following command to check the health of kafka-controller from SSP installer as root.
  
  k get pods -n nsxi-platform | grep kafka-controller
  
  
• If kafka is unhealthy, contact Broadcom support.
• If kafka is healthy, check for SSL handshake error between IDPS agent and kafka broker. Keywords to look for – 'SSL handshake failed' and 'Nsxi: Kafka broker(s) are down.' in /var/log/nsx-syslog. If there is an SSL handshake error, restart proton on the node where the service COMMON_AGENT_SERVICE is the leader. You can determine where this service is the leader by issuing the 'get cluster status verbose' command on the NSX Manager.
  
  
• Check if there is a network policy preventing communication between the transport node where the alarm is seen and the kafka broker.
  
  k get svc -n nsxi-platform | grep external
  
  
• Check connectivity from affected host to all the kafka external IPs.
  
  nc -vz <messaging IP> 9092

Environment

VMware NSX 4.2.1

NAPP 4.2

SSP 5.0 and above

Cause

• "Message Transmission Failed" alarm is raised by the IDPS client. The rdkafka library is used to detect whether the communication channel to the Kafka broker is down.
• This issue is related to the Kafka broker and the rdkafka library used to monitor the connection. The alarm is being incorrectly triggered because the rdkafka library marks the channel as down, even though the transmission is proceeding without issue. Essentially, the alarm itself is misleading and does not reflect a functional problem with the system.

Resolution

• There is no functional impact on your system. The alarm may cause confusion or concern, but it does not indicate any issue with the actual data flow or security event processing. To recover from this alarm, simply clear it from the Manager UI. Once cleared, there should be no further disruptions.
• This is issue fixed in NSX 4.2.2 and 9.0.0