NAPP is in 'Degraded' state, and the NDR is repeatedly going DOWN.

Products

VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

The NDR pipeline crashes intermittently due to improper handling of HTTP requests containing an excessively long Host header triggered by signature ID 1121481, which detects attempts to exploit CVE-2001-0241.

Environment

Prod Environment (NAPP 4.2.0.0 and NSX- 3.2.3.1)

Cause

The detection signature 1121481 for CVE-2001-0241 triggers on HTTP requests with an overly long Host header. The pipeline fails to process these messages correctly, leading to a crash. Kafka retains these malformed messages, causing repeated crashes when the worker service consumes them.

(CVE-2001-0241 is a buffer overflow vulnerability in Internet Printing ISAPI extension in Windows 2000. An unauthenticated remote attacker can exploit this vulnerability and execute arbitrary code by sending a specially crafted HTTP request to the affected system.)

Accessing Logs for Specific Pods on NSX Manager CLI

To retrieve logs for a specific pod, follow the steps below:

Log in to the NSX Manager CLI
SSH the NSX Manager Command Line Interface (CLI) using the root credentials.
Identify the Target Pod
Run the following command to list the pods under the nsxi-platform namespace and filter for the desired pod:

napp-k get pods -n nsxi-platform | grep <POD-Name>

Example:

root@nsx-manager:~# napp-k get pods -n nsxi-platform | grep kafka kafka-0 1/1 Running 0 4h14m kafka-1 1/1 Running 0 32h kafka-2 1/1 Running 0 32h
Retrieve Logs for the Specified Pod
Once the target pod is identified, use the following command to view its logs:

napp-k logs <POD-Name> -n nsxi-platform | less

Example:

To check logs for kafka-2, use:

root@nsx-manager:~# napp-k logs kafka-2 -n nsxi-platform | less

The less utility allows you to navigate through the log output efficiently.

Relevant Logs:

Error 1: Kafka SSL Handshake Failures

kafka-0 log

log":"2025-02-03T11:25:42.041072756Z stdout F \"\"2025-02-03T11:25:42.040ZGMT INFO data-plane-kafka-network-thread-0-ListenerName(EXTERNAL)-SSL-8 Selector - [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /192.168.4.1 (channelId=192.168.3.191:9092-192.168.4.1:24533-710623) (SSL handshake failed)","kubernetes":{"pod_name":"kafka-0","namespace_name":"nsxi-platform","pod_id":"e4ea9db2-92bb-4fcf-a918-5fdf78fb665d","host":"dc3-prod-napp-worker-nodepool-a1-7l446-5876b5dfc8xxj9nf-c5l46","container_name":"kafka","docker_id":"758313eec6b1cdba5458c78ce1dab9d3919e891e0995f89a5c5ed66ae0822bdd","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/third-party/kafka@sha256:a0016b1960a4968d00b9936ee541cc1c82ab7666a3a17edce26d3ad458c74da4","container_image":"sha256:e1c21cb5eb57ed06679ac49a86f97061d38e922a1cfb9e5326c046b2918ca4bd"}}

Error 2: NTA Alert with No Data Flows

nsx-ndr-worker-nta-event-translator log

{"log":"2025-03-19T05:25:19.537166563Z stderr F 2025-03-19 05:25:19,536 - nsx_kafka_utils.threaded_consumer - ERROR - Rejecting invalid message on topic ndr_nta_event(0): Handling of NtaAlert with no data flows is not supported. The detector name: UNCOMMONLY_USED_PORT. (first 1kb of message, b64 encoded: CiQ5ZTIyODQyZS1hNjE2LTQ3NDYtYWU1MS0wYTE3OTRiODIwNTYQBBoUVU5DT01NT05MWV9VU0VEX1BPUlQgu7+o3toyKLu/qN7aMjoECA8QFEIASrMGClQKFU1FVEFEQVRBX0tFWV9IT1RTUE9UUxI7EjkKEUlQLTEwLjI0OC4xNDguMTQxCiQ1MDIyZGFlZS1jOGY4LTJkMjYtMWUyMy1hZjVlZjY2ODE1ODcKvgEKBlJFUE9SVBKzASKwAXsicmVwb3J0X3R5cGUiOiAiVU5DT01NT05MWV9VU0VEX1BPUlQiLCAiZXhwZWN0ZWRfcG9ydCI6IFsxMzVdLCAiZGlzY292ZXJlZF9wb3J0IjogNDk4MDEsICJhcHBfaWQiOiA1MDksICJhcHBfbmFtZSI6ICJEQ0VSUEMiLCAicHJvdG9jb2wiOiAiVENQIiwgInZ1bG5lcmFibGVfYXBwX3ZlcnNpb24iOiAiMCJ9Ch4KEU1FVEFEQVRBX0tFWV9QT1JUEgkSBwoFNDk4MDEKMgoZTUVUQURBVEFfS0VZX0RTVF9DT01QVVRFUxIVEhMKEUlQLTEwLjI0OC4xNDguMTQxCkUKGU1FVEFEQVRBX0tFWV9TUkNfQ09NUFVURVMSKBImCiQ1MDIyZGFlZS1jOGY4LTJkMjYtMWUyMy1hZjVlZjY2ODE1ODcKIAoVTUVUQURBVEFfS0VZX1BST1RPQ09MEgcSBQoDVENQCioKFE1FVEFEQVRBX0tFWV9EU1RfSVBTEhISEAoOMTAuMjQ4LjE0OC4xNDEKKQoUTUVUQURBVEFfS0VZX1NSQ19JUFMSERIPCg0xMC4yMzguMjMwLjE3CisKEWV2ZW50X2Rlc2NyaXB0aW9uEhYiFFVOQ09NTU9OTFlfVVNFRF9QT1JUCtgBCgpWSVpfRklMVEVSEskBIsYBeyJjb25kaXRpb24iOiAiQU5EIiwgInZpekZpbHRlcnMiOiBbeyJrZXkiOiAiRkxPV19EU1RQT1JUIiwgInZhbHVlcyI6IFsiNDk4MDEiXSwgImNyaXRlcmlhIjogIkVRIiwgImNvbmRpdGlvbiI6ICJPUiJ9LCB7ImtleSI6ICJGTE9XX0FQUElEIiwgInZhbHVlcyI6IFsiNTA5Il0sICJjcml0ZXJpYSI6ICJFUSIsICJjb25kaXRpb24iOiAiT1IifV19UhNDT01NQU5EX0FORF9DT05UUk9MWhFOT05fU1RBTkRBUkRfUE9SVGoA)","kubernetes":{"pod_name":"nsx-ndr-worker-nta-event-translator-5f75d9c67d-r9msl","namespace_name":"nsxi-platform","pod_id":"a0203458-fe2f-4f31-9b2f-92db6f8d2397","host":"dc3-prod-napp-worker-nodepool-a1-7l446-5876b5dfc8xxj9nf-jh24j","container_name":"worker","docker_id":"53c71cdcabd0e3d54c93328eb6a40c32b3ebba86fe83d60308f11b1446055240","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker@sha256:53327d5103359ab881d29aafe1255628e2d6ac6ebecf94bf9c923a3a60968f47","container_image":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker:332__releaselongview4.2.16-041ff729.focal"}}

{"log":"2025-03-19T05:25:19.537206137Z stderr F nsx_kafka_utils.errors.InvalidMessage: Handling of NtaAlert with no data flows is not supported. The detector name: UNCOMMONLY_USED_PORT.","kubernetes":{"pod_name":"nsx-ndr-worker-nta-event-translator-5f75d9c67d-r9msl","namespace_name":"nsxi-platform","pod_id":"a0203458-fe2f-4f31-9b2f-92db6f8d2397","host":"dc3-prod-napp-worker-nodepool-a1-7l446-5876b5dfc8xxj9nf-jh24j","container_name":"worker","docker_id":"53c71cdcabd0e3d54c93328eb6a40c32b3ebba86fe83d60308f11b1446055240","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker@sha256:53327d5103359ab881d29aafe1255628e2d6ac6ebecf94bf9c923a3a60968f47","container_image":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker:332__releaselongview4.2.16-041ff729.focal"}}

Error 3: server_compute_fqdn Field Exceeds Max Length

nsx-ndr-worker-detection-event-aggregator log

{"log":"2025-03-19T15:21:25.770972299Z stderr F nsx_kafka_utils.errors.WorkerThreadError: detection_event_flow.server_compute_fqdn exceeds max length 255","kubernetes":{"pod_name":"nsx-ndr-worker-detection-event-aggregator-69654f8d58-dlgrq","namespace_name":"nsxi-platform","pod_id":"67f0ea17-d1b5-4df2-a4ac-6671602b1391","host":"dc3-prod-napp-worker-nodepool-a1-7l446-5876b5dfc8xxj9nf-r5mv4","container_name":"worker","docker_id":"f3952515bc53bd343f4c1a0f30e0c0ed4d67150579c2f9e936972dab7522b7e1","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker@sha256:53327d5103359ab881d29aafe1255628e2d6ac6ebecf94bf9c923a3a60968f47","container_image":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker:332__releaselongview4.2.16-041ff729.focal"}}

Error 4: Failed SIEM Notification - 401 Unauthorized

nsx-ndr-worker-siem-notification-sender log

{"log":"2025-03-19T11:06:38.700135143Z stderr F 2025-03-19 11:06:38,699 - nsx_ndr_service.siem.siem_sender - ERROR - Failed to send SIEM event notification: HTTP Error: 401 Unauthorized - Response Body: {\"text\":\"Invalid authorization\",\"code\":3}","kubernetes":{"pod_name":"nsx-ndr-worker-siem-notification-sender-7d956cfbb-7n8w5","namespace_name":"nsxi-platform","pod_id":"4a5e0c90-1d96-4146-b998-784ce28cec4d","host":"dc3-prod-napp-worker-nodepool-a1-7l446-5876b5dfc8xxj9nf-r5mv4","container_name":"worker","docker_id":"4af79a71d151a853d2129530c5b62247219df9cec55c4f44f57dbf8f494c6710","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker@sha256:53327d5103359ab881d29aafe1255628e2d6ac6ebecf94bf9c923a3a60968f47","container_image":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker:332__releaselongview4.2.16-041ff729.focal"}}

{"log":"2025-03-19T11:09:07.761781957Z stderr F raise errors.FailedToSendSiemNotification(error_msg) from e","kubernetes":{"pod_name":"nsx-ndr-worker-siem-notification-sender-7d956cfbb-7n8w5","namespace_name":"nsxi-platform","pod_id":"4a5e0c90-1d96-4146-b998-784ce28cec4d","host":"dc3-prod-napp-worker-nodepool-a1-7l446-5876b5dfc8xxj9nf-r5mv4","container_name":"worker","docker_id":"4af79a71d151a853d2129530c5b62247219df9cec55c4f44f57dbf8f494c6710","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker@sha256:53327d5103359ab881d29aafe1255628e2d6ac6ebecf94bf9c923a3a60968f47","container_image":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker:332__releaselongview4.2.16-041ff729.focal"}}
{"log":"2025-03-19T11:09:07.761784042Z stderr F nsx_ndr_service.siem.errors.FailedToSendSiemNotification: HTTP Error: 401 Unauthorized - Response Body: {\"text\":\"Invalid authorization\",\"code\":3}","kubernetes":{"pod_name":"nsx-ndr-worker-siem-notification-sender-7d956cfbb-7n8w5","namespace_name":"nsxi-platform","pod_id":"4a5e0c90-1d96-4146-b998-784ce28cec4d","host":"dc3-prod-napp-worker-nodepool-a1-7l446-5876b5dfc8xxj9nf-r5mv4","container_name":"worker","docker_id":"4af79a71d151a853d2129530c5b62247219df9cec55c4f44f57dbf8f494c6710","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker@sha256:53327d5103359ab881d29aafe1255628e2d6ac6ebecf94bf9c923a3a60968f47","container_image":"projects.registry.vmware.com/nsx_application_platform/clustering/nsx-ndr-worker:332__releaselongview4.2.16-041ff729.focal"}}

Resolution

The issue can be temporarily mitigated by disabling IDS signature 1121481, which detects exploitation attempts related to a legacy vulnerability (CVE-2001-0241) targeting Windows 2000 systems. Disabling this signature will prevent detection and blocking of such attempts; however, given the age of the vulnerability and the unlikelihood of affected systems being present in the protected network, this poses minimal risk. The detections appear to result primarily from vulnerability scans—either legitimate internal assessments or reconnaissance by potential attackers, rather than active exploitation attempts.

Furthermore, this specific signature has been removed from the IDS signature bundle distributed across all product versions, ensuring that this issue will no longer be encountered. Although similar incidents could theoretically occur due to other signatures until the planned fix in version 5.1 is implemented, this specific problem has been resolved in the current signature set.