Excessive SSL / Kafka Error Logs in NSX logs (nsxapi.log) After SSP FIPS Onboarding
Article ID: 419655
Updated On:
Products
Issue/Introduction
Seeing kafka-producer-network-thread thread disconnected excessively in nsxapi log after onboarding to FIPS enabled SSP.
Below Errors will be observed in NSX manager /var/log/proton/nsxapi.log
2025-11-19T19:23:24.119Z INFO kafka-producer-network-thread | producer-8 Selector 142943
[Producer clientId=producer-8] Failed re-authentication with 190.x.y.z (channelId=-1)
(Failed to process post-handshake messages)
2025-11-19T19:23:24.119Z INFO kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Node -1 disconnected.
2025-11-19T19:23:24.119Z ERROR kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Connection to node -1 (190.x.y.z:9092) failed authentication due to:
Failed to process post-handshake messages
2025-11-19T19:23:24.119Z INFO kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Cancelled in-flight API_VERSIONS request with correlation id 63861
due to node -1 being disconnected (elapsed time since creation: 3ms, elapsed time since send: 3ms,
request timeout: 30000ms)
2025-11-19T19:23:24.119Z WARN kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Bootstrap broker 190.x.y.z:9092 (id: -1 rack: null) disconnected
Environment
SSP 5.1.1
NSX version where this is known issue: 4.2.0, 4.2.1.1, 4.2.1.2
Cause
NSX contains logic to generate self signed certificates for CommonAgent and PaceAgent that are used by the agents to connect to SSP and communicate state. However, these generated self signed certificates do not include a FIPS required extension needed for SSP FIPS , ExtendedKeyUsage with a value of "cientAuth". We observed that Kafka rejected these certificates because it appears that the ExtendedKeyUsage is reflecting "serverAuth".
To help with this self signed certificate is created by SSP and during onboarding the existing kafka client certificates are replaced.
There is an existing bug in common agent which leaves stale threads
Because of this we would be seeing the following logs for extended period of time and possibly a couple of 100,000 log lines filling up nsxapi log.( /var/log/proton/nsxapi.log )
2025-11-19T19:23:24.119Z INFO kafka-producer-network-thread | producer-8 Selector 142943
[Producer clientId=producer-8] Failed re-authentication with /190.x.y.z (channelId=-1)
(Failed to process post-handshake messages)
2025-11-19T19:23:24.119Z INFO kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Node -1 disconnected.
2025-11-19T19:23:24.119Z ERROR kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Connection to node -1 (/190.x.y.z:9092) failed authentication due to:
Failed to process post-handshake messages
2025-11-19T19:23:24.119Z INFO kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Cancelled in-flight API_VERSIONS request with correlation id 63861
due to node -1 being disconnected (elapsed time since creation: 3ms, elapsed time since send: 3ms,
request timeout: 30000ms)
2025-11-19T19:23:24.119Z WARN kafka-producer-network-thread | producer-8 NetworkClient 142943
[Producer clientId=producer-8] Bootstrap broker 190.x.y.z:9092 (id: -1 rack: null) disconnected
We would see the above block every second.
Resolution
To stop seeing these logs we need to do rolling restart of proton.
Please contact Broadcom support for further assistance on this.
Feedback
