SSP Infrastructure Sync status is DOWN, Inventory Sync status is Unknown and site status shows for NSX as Not Ready
search cancel

SSP Infrastructure Sync status is DOWN, Inventory Sync status is Unknown and site status shows for NSX as Not Ready

book

Article ID: 434265

calendar_today

Updated On:

Products

VMware vDefend Firewall with Advanced Threat Prevention VMware vDefend Firewall

Issue/Introduction

NSX is in a Not Ready state if the infrastructure sync is DOWN and Inventory Sync is Unknown.

Possible Scenarios

This issue has been observed in the following scenarios:

  • Brownfield case: Upgrade of NSX 9.0.1 to 9.0.2 which is already onboarded to SSP.

  • Brownfield case: Upgrade of NSX 4.2.1 to 4.2.3 which is already onboarded to SSP.

  • Greenfield case: Onboarding of an affected NSX version to SSP 5.1.0/5.1.1.

  • Network / Security Policy configuration: Strict App-ID, application-aware filtering (e.g., Palo Alto upstream firewalls), or deep packet inspection rules applied to the messaging port (9092) instead of standard port-based configurations.

Symptoms

Example 1

After logging into the SSPI installer and describing the site, you can see COMMON_INIT_COMPLETE failed:

k describe site -n nsxi-platform

Output:

status:
  conditions:
  - lastTransitionTime: "2026-02-09T21:11:57Z"
    message: COMMON_INIT_COMPLETE failed
    reason: AgentInitNotReady
    status: "False"
    type: CommonAgentReady

Example 2

After logging into the SSPI installer and describing the site, you can see COMMON_FULLSYNC failed:

- lastTransitionTime: "2026-05-27T17:20:08Z"
  message: 'COMMON_FULLSYNC failed due to: java.lang.Exception: produceCertMsgs'
  reason: FullSyncNotReady
  status: "False"
  type: CommonAgentReady
currentState: NotReady
message: OnboardingInProgress


An OpenSSL connectivity test from the NSX Manager node to the SSP messaging server on port 9092 fails during the TLS handshake phase with a socket reset error, even though basic Layer 4 ping/netcat utilities show the port is reachable:

openssl s_client -connect <ssp-messaging-fqdn>:9092 -servername <ssp-messaging-fqdn>

Output:

CONNECTED(00000004)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 317 bytes
Verification: OK

 

NSX Manager UI Verification

  • Path: System > Certificates

  • Symptom: When filtering for "NAPP", the certificates NAPP_COMMON_AGENT, NAPP_METRICS_AGENT, and NAPP_PACE_AGENT show a count of 0 under the Used By column.

Environment

vDefend Security Services Platform(SSP) version: 5.1.0, 5.1.1

NSX versions where this is known issue: 4.2.0, 4.2.1.1, 4.2.1.2, 4.2.2 , 4.2.3, 9.0.0, 9.0.1, 9.0.2

Cause

This condition can be triggered by either of the following two root causes:

  1. Internal Race Condition: A race condition occurred between the Common Agent and the Trust Management service on the NSX Manager. While the system correctly identified that a certificate was needed (Marker = True), the specific internal service responsible for generating that certificate was in the middle of a "leadership election." Because this service only acts on requests when it is the confirmed leader, the request to create the certificate was dropped and never retried.

  2. Upstream Network Policy Interception: The upstream firewall is configured using application-aware signatures (App-ID/App Name matching kafka or ssl) instead of standard port-based definitions. During onboarding, Kafka streams non-HTTP binary payloads inside a TLS envelope over port 9092. If the firewall fails to recognize this traffic as standard HTTPS or classifies it strictly by application filters, it drops or strips the TLS handshake packets, causing an immediate TCP connection reset (errno=104) on the destination broker.

Logs

SSP Site-Service Logs: (k -n nsxi-platform logs <site-service-pod>)

2026-03-11T14:58:00.758Z INFO reconcileNsxData patch {"request": {"name":"<UUID>","namespace":"nsxi-platform"}, "reconcileID": "<UUID>", "changed": true, "newSite": {"currentState":"NotReady","message":"OnboardingInProgress","nsxInfo":{"serviceNameRef":"nsx-<UUID>","version":"9.0.2.0.25150386","clusterStatus":"Degraded","nsxClusterID":"<UUID>","formFactor":"Medium"},"conditions":[{"type":"CertificatesInSync","status":"True","lastTransitionTime":"2026-03-10T16:26:02Z","reason":"CertificatesInSync","message":""},{"type":"ConnectionEstablished","status":"True","lastTransitionTime":"2026-03-10T16:26:02Z","reason":"ConnectionEstablished","message":""},{"type":"SiteConditionConfiguredPlatformDeploymentConfig","status":"False","lastTransitionTime":"2026-01-02T16:23:48Z","reason":"NotRequired","message":""},{"type":"CreatedNsxSspApplianceInfo","status":"True","lastTransitionTime":"2026-01-02T16:24:48Z","reason":"Created","message":""},{"type":"NsxStreamingReady","status":"True","lastTransitionTime":"2026-01-02T16:26:12Z","reason":"NsxConfigTOIUpdated","message":""},{"type":"CommonAgentReady","status":"False","lastTransitionTime":"2026-02-09T21:11:57Z","reason":"CommonAgentCertProfileNotReady","message":"COMMON_CERT_PROFILE_READY failed"},{"type":"PaceAgentReady","status":"True","lastTransitionTime":"2026-02-09T17:03:14Z","reason":"Ready","message":""},{"type":"IdsEnabled","status":"True","lastTransitionTime":"2026-01-30T10:32:14Z","reason":"IdsIsEnabled","message":""}]}}
2026-03-11T14:58:11.743Z ERROR Reconciler error {"request": {"name":"<UUID>","namespace":"nsxi-platform"}, "reconcileID": "<UUID>", "error": "subreconciler reconcileNappApplianceInfo failed: failed to filter agent status: agent AGENT_TYPE_COMMON is not enabled"}

Example 1: NSX Manager Logs (/var/log/proton/nsxapi.log)

2026-03-11T14:58:06.595Z ERROR kafka-producer-network-thread | producer-533 NetworkClient 6021 [Producer clientId=producer-533] Connection to node -1 (/<IP_ADDRESS>:9092) failed authentication due to: SSL handshake failed
2026-03-11T15:01:39.624Z ERROR GMLE-Leadership-Executor CommonAgentServiceImpl 6021 NAPP [nsx@4413 comp="nsx-manager" errorCode="MP1" level="ERROR" s2comp="CommonAgent" subcomp="manager"] Certificates are not ready
2026-03-11T15:01:39.635Z INFO GMLE-Leadership-Executor StatusTrackingServiceImpl 6021 INTELLIGENCE [nsx@4413 comp="nsx-manager" level="INFO" subcomp="manager"] updateAction COMMON_INIT_COMPLETE state ERROR

Example 2: NSX Manager Logs (/var/log/proton/nsxapi.log)

The logs show repeated failures to send data to the platform and Kafka timeout errors:

WARN CommonAgentDeltaProcessor CommonAgentKafkaClient [nsx@6876 comp="nsx-manager" level="WARNING" s2comp="CommonAgent" subcomp="manager"] Failed to send data to K8S platform during attempt 3
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic intelligence_platform not present in metadata after 60000 ms.
ERROR intelligence-message-processor IntelligenceNsxCommunicationServiceImpl [nsx@6876 comp="nsx-manager" errorCode="PM91905" level="ERROR" subcomp="manager"] Failed to send config message updates to NSX Intelligence.
WARN kafka-producer-network-thread | producer-60 NetworkClient [Producer clientId=producer-60] Co

Resolution

 

Path A: Verifying and Modifying Upstream Firewall Rules

Before executing backend database modifications, verify if network security policies are dropping the TLS traffic on management ports.

  1. Review the upstream firewall configurations (e.g., Palo Alto security policies) managing the traffic between the NSX Manager nodes and the SSP cluster.

  2. Check if the rules utilize strict App-ID tracking (such as identifying application profiles for kafka or ssl) instead of static port profiles.

  3. Remediation: Create or modify the security rules to allow traffic over explicit destination service ports; specifically defining TCP 9092 and TCP 443 explicitly by port number rather than relying strictly on Application names. Commit the policy changes and verify if onboarding automatically finishes.

Path B: Adjusting the Corfu Database State

If network paths are verified and open, this resolution requires updating the Corfu database and restarting the proton service on the nodes currently holding leadership for both the CertificateShardingServiceImpl and the CommonAgent.

Because these steps involve manual modifications to the Corfu database, we strongly recommend contacting Broadcom Support via a support ticket to assist with the execution.

This issue is tracked internally via the reference #3670505.

Powered by Wolken Software