Kafka Connectivity Issue Between NSX Config Agent and NAPP Triggering 'Config Agent Unhealthy' Alarm"

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

If an Alarm with  Feature "NSX Application Config Agent" and Event Type "Config Agent Unhealthy" is seen on NSX UI triggered due to connection issue in the channel between NSX Config Agent and NSX Application Platform(NAPP)

Impact: This will result in stale or missed configuration updates on NAPP due to which 

Metrics API calls made with intent path may return error response for certain queries. Sample error response: {"error_code":950010,"module_name":"Nsx-Metrics","error_message":"API Validation error : Resource type <resource_type> does not match resource id(s) : [resource_ids]"}

Intelligence could get impacted with latest configs not reflected in now view.

Environment

NSX 4.2.1, NSX 4.2.1.2

Cause

Kafka Connectivity between NSX Config Agent to NAPP is broken and unable to recover from a bad state.

 

To confirm the alarm's root cause, access the affected NSX Manager's(To get affected NSX Manager, expand alarm and check 'Reported by Node') command-line interface and execute the following command.

cd /var/log/proton

zgrep "Exception happened when receiving instruction from NSX Intelligence" -A 1 nsxapi.*

Above command should show many occurrences of below error:

java.lang.IllegalStateException: This consumer has already been closed.

Resolution

Restart Proton service on the affected NSX Manager.
Restart of proton can be done by executing below command on affected NSX Manager command-line interface

systemctl restart proton

 

If restarting proton is not feasible please contact Broadcom Support Team