Troubleshooting NSX Manager Onboarding and Site Connectivity Issues in SSP

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

When onboarding NSX Manager into the Security Services Platform (SSP), you may see site readiness problems (Readiness: Not Ready), connectivity issues (Inventory Sync: Down / Unknown), or a top-of-screen banner “No valid license available.”

These symptoms are usually caused by one or more of the following categories:

(1) Certificate / trust mismatches (expired or unsynced certs).

(2) Network/firewall blocking of messaging ports.

(3) Stale NSX platform registry or kubeconfig entries (leftover from undeploys).

(4) Agent concurrency or sync-thread issues inside NSX or

(5) Inability to reach external helm/repos during feature activation.

The most effective troubleshooting approach is to identify the symptom, verify certs & ports, check site/service pod health, inspect relevant NSX logs and then apply the targeted resolution steps below.

Environment

vDefend Security Services Platform 5.0, 5.1

Resolution

 

Symptoms & How to Verify (Run from SSP Installer or NSX Manager)	KB Reference
### Symptom 1: Unknown Authority Certificate Failure UI Manifestation: NSX Manager Tab shows Readiness: Not Ready, Inventory Sync: Down, Infrastructure Sync: Up. How to Verify: 1. Log into the NSX Manager CLI and verify basic messaging port connection: nc -vz <ssp-messaging-url> 9092 2. Log into the SSP Installer VM, identify the site-service pod, and check its logs: k get pods -n nsxi-platform | grep site-service k logs <site-service-pod> -n nsxi-platform | grep -E "x509|unknown authority"	KB 407753
### Symptom 2: Site Onboarding Failure Due to Expired Internal Trust Cert UI Manifestation: Onboarding fails explicitly with failed to connect to site – x509: certificate signed by unknown authority. How to Verify: Log into the SSP Installer VM and inspect the full description of the site configuration object: k describe site -n nsxi-platform Look for the specific status block error: {"error_code":940008, "error_message":"Supplied certificate data is invalid because: Certificate has expired."}	KB 405890
### Symptom 3: Kubeconfig Validation / API Connection Timeout UI Manifestation: SSP UI Banner displays No valid license available. System tab displays Infra/Inventory Sync = Unknown. How to Verify: Log into the NSX Manager CLI and search the API process manager logs for network timeouts connecting back to the workload cluster over port 6443: grep -i "Error while validating kubeconf" /var/log/proton/nsxapi.log Example Log Match: CloudNativeKubeconfigServiceImpl ... Unable to connect to the server: dial tcp <ssp-ip>:6443: i/o timeout	KB 413368
### Symptom 4: CommonAgent Delta Processing NullPointerException UI Manifestation: No Valid License Error banners triggered shortly after initiating an NSX onboarding operation. How to Verify: 1. In the NSX Manager UI, navigate to System > Fabric > Hosts and check if any underlying host clusters are degraded. 2. Log into the NSX Manager CLI and target the NAPP CommonAgent executor context: grep -A 5 "java.lang.NullPointerException" /var/log/proton/nsxapi.log | grep -E "CommonAgentDeltaProcessor|buildCertUpdateMessage"	KB 396403
### Symptom 5: Concurrent Database Update / Transaction Conflict Failure UI Manifestation: Feature activation fails completely. The SSP UI shows No Valid License Available and Infra Sync: DOWN. How to Verify: Log into the NSX Manager CLI and inspect the transaction engine outputs for parallel execution conflicts during onboarding operations: grep -i "ConcurrentUpdateException" /var/log/proton/nsxapi.log Example Log Match: Caught exception in ScheduledExecutorService.com.vmware.nsx.management.container.exceptions.ConcurrentUpdateException: Operation failed because of conflicting transaction.	KB 390413
### Symptom 6: Helm Repository Ingress / Repository Reachability Failure UI Manifestation: Feature activation stops mid-process due to a Helm chart repo failure returning a BAD_REQUEST (Error Code: 46011). How to Verify: Log into the SSP Installer VM and inspect the real-time reconciler event logging engine inside the site-service deployment: k logs deploy/site-service -n nsxi-platform | grep -i "Helm add repo operation failed" Identify one of three trace states: 1. DNS/Network Unreachable: lookup projects.registry.vmware.com ... server misbehaving 2. Stale/Missing Non-OCI Repo: failed to fetch ... index.yaml : 404 3. Registry TLS Handshake Error: tls: failed to verify certificate: x509: certificate signed by unknown authority	KB 393412
### Symptom 7: Stale NAPP Meta Registration Conflict Post-Undeployment UI Manifestation: SSP UI displays Readiness: Not Ready, Infra/Inventory Sync: Unknown. No explicit stack trace errors display within the active site-service pod logs (returns normal 200 OK calls). How to Verify: Log into the SSP Installer VM and investigate the interoperability flags inside the site specifications template: k describe site -n nsxi-platform Confirm condition failure state: Type: SiteConditionConfiguredPlatformDeploymentConfig, Status: False, Reason: RequiredForInterop	KB 413561
### Symptom 8: Missing On-Disk CommonAgent Keystore Asset (NoSuchFileException) UI Manifestation: SSP UI displays Readiness: Not Ready and Infra Sync: Down. Features like Intelligence, NDR, and Malware Prevention are activated but fail to connect. How to Verify: 1. From the SSP Installer, describe the cluster site data to identify full-sync readiness states: k describe site -n nsxi-platform | grep -E "COMMON_FULLSYNC|FullSyncNotReady" 2. On the NSX Manager CLI, check for standard filesystem read errors on the background keystore folder paths: grep -A 3 "NoSuchFileException" /var/log/proton/nsxapi.log Example Log Match: java.nio.file.NoSuchFileException: /home/secureall/secureall/.store/.commonagent_keystore	KB 409676
### Symptom 9: Proton OOM Service Crash Loop via Massive Directory User Sync UI Manifestation: Continuous execution restarts of the Proton engine on NSX Manager nodes occurring within minutes/hours of onboarding an LDAP-enabled SSP instance. How to Verify: 1. Log into the NSX Manager CLI and view the Tomcat runtime status wrapper engine for explicit out-of-memory logs: grep -A 5 "The JVM has run out of memory" /var/log/proton/proton-tomcat-wrapper.log 2. Calculate the active count of AD user records loaded in the underlying Corfu DB to see if it exceeds 100,000 strings: /opt/vmware/bin/corfu_tool_runner.py -n nsx -t DirectoryUser -o showTable | wc -l	KB 403824
### Symptom 10: Duplicate Site UUID Registration Error during Onboarding/Re-onboarding UI Manifestation: Offboarding functions hang indefinitely or subsequent onboarding operations error out with an explicit message: This site is already registered to an SSP or NAPP instance. How to Verify: On-board the NSX Manager in SSP and UI would throw this error.	KB 382295
### Symptom 11: PaceAgent Subscription Processing Pipeline Stalled (PACE_UFOSTORE_SUBSCRIBE) UI Manifestation: Connectivity Agent shows Infra Sync: Down under view details on the SSP UI, while the site configuration shows an unready state without active alarms. How to Verify: 1. Log into the SSP Installer VM and inspect the condition tables on the cluster object record: k describe site -n nsxi-platform Verify condition block error state: Type: PaceAgentReady, Status: False, Message: PACE_UFOSTORE_SUBSCRIBE not started 2. Grab the live site-service container outputs to see the state delta tracking loop: k logs deploy/site-service -n nsxi-platform | grep -i "PACE_UFOSTORE_SUBSCRIBE"	KB 402380
### Symptom 12: Missing Transport Node Kafka Message Acknowledgements (raw_flow Dropped) UI Manifestation: Infra Sync: Down on the SSP, full/delta synchronizations show failed, and network telemetry flows are missing on the Intelligence dashboard. How to Verify: 1. Run log extraction on the NSX Manager CLI to check for active produceCertMsgs NullPointerExceptions: grep -A 10 "produceCertMsgs" /var/log/proton/nsxapi.log grep -C 10 "Transport-node UUIDs not found" /var/log/proton/nsxapi.log grep -C 10 "Node/status UUIDs with missing status" /var/log/proton/nsxapi.log 2. Log into an affected Transport Node (ESXi Host) CLI and verify telemetry flow acknowledgment metrics: nsxcli -c get intelligence flow stats ack Identify if total acknowledged packets (Total Ack'ed) stop incrementing compared to Total Sent for the raw_flow topic.	KB 401182
### Symptom 13: Keystore Generation Stalled Post NSX Backup Restore Execution UI Manifestation: Post-restore of NSX Manager nodes, the SSP connection is persistently lost. NSX Manager UI outputs active platform health alerts, while the SSP UI lists the manager as Not Ready. How to Verify: Log into the NSX Manager CLI and review the platform bootstrap logs to confirm if the system failed to reconstruct internal security keystore objects on local storage: grep -i "NoSuchFileException.*_keystore" /var/log/proton/nsxapi.log	KB 390751
### Symptom 14: Continuous Proton Restart Loop due to GMLE Leadership Lease Loss Timeout UI Manifestation: Infra Sync: Down inside the SSP UI. The Proton application service on the NSX Manager crashes and restarts endlessly. How to Verify: Log into the NSX Manager CLI and correlate the Tomcat manager application restart log with the core platform system logs: grep "APPLICATION IS GOING RESTART" /var/log/proton/proton_restart.log grep "Lease loss acknowledgement has not been received" /var/log/proton/nsxapi.log Look for timeouts related specifically to the COMMON_AGENT_SERVICE process thread context.	KB 414655
### Symptom 15: Inventory Sync Recovery Stalled by Corfu Cache / Stream Timeouts (PACE_AGENT_INIT) UI Manifestation: Inventory Sync switches to Down during onboarding and fails to self-recover or transition to active after 40+ minutes of run-time. How to Verify: Log into the NSX Manager CLI and check the Status Tracking service logs to see if the platform pace synchronization mechanism is looping on execution tracking errors: grep -A 3 "PACE_AGENT_INIT" /var/log/proton/nsxapi.log Example Log Match: action_name: "PACE_AGENT_INIT" required_for_init: true status: ERROR message: "PACE_AGENT_INIT failed"	KB 412043
### Symptom 16: Interlocking Site De-registration Re-onboarding Lockout State UI Manifestation: When re-registering an NSX node, the UI fails with: The site being reconnected is still being offboarded. Please allow time for offboarding to complete before reconnecting. How to Verify: While onboarding NSX Manager to SSP, UI Displays: The site being reconnected is still being offboarded. Please allow time for offboarding to complete before reconnecting.	KB 427080
### Symptom 17: Port 9092 TLS Handshake Reset via Upstream Application-Aware Firewall Policies UI Manifestation: Infra Sync: Down, Inventory Sync: Unknown, Site status is Not Ready. (Observed during NSX version upgrades or greenfield setups behind firewalls). How to Verify: 1. From the SSP Installer, check for initialization failure states inside the cluster site metadata: k describe site -n nsxi-platform | grep -E "COMMON_INIT_COMPLETE failed|COMMON_FULLSYNC failed" 2. Run an OpenSSL connectivity probe from the NSX Manager CLI directly to the Kafka messaging endpoint over port 9092: openssl s_client -connect <ssp-messaging-fqdn>:9092 -servername <ssp-messaging-fqdn> Confirm if L4 ping/netcat works, but the OpenSSL TLS handshake receives an ungraceful reset: write:errno=104 / no peer certificate available.	KB 434265
### Symptom 18: NSX TOI Configuration Update Schema Pipeline Reconciler Failure UI Manifestation: Site readiness displays Unknown error while configuring site. Inventory and Infrastructure fields evaluate to an Unknown status string. How to Verify: Log into the SSP Installer VM and evaluate the logging state of both the configuration manager pod and the site-service deployment runtime log strings: k logs deploy/nsx-config -n nsxi-platform | grep -i "Exception when sending TOI update" k logs deploy/site-service -n nsxi-platform | grep -A 2 "NsxConfigTOIUpdateFailed" Example Log Match: NSX TOI Update error: Internal server error has occurred. (error_code: 900069)	KB 440678
### Symptom 19: Concurrent Batch Certificate Lockout via Stale Host Onboarding Tasks UI Manifestation: Both Inventory Sync and Infrastructure Sync engines stay stuck in a Down state continuously within SSP 5.1.1 environments. How to Verify: Log into the NSX Manager CLI and pull error codes indicating database schema write locks preventing security token or key rotations: grep -i "conflicting operations are running" /var/log/proton/nsxapi.log Example Log Match: A certificate batch replace operation cannot be started at this time because conflicting operations are running: TRANSPORT_NODE_ONBOARDING_IN_PROGRESS. (Error code: 2190)	KB 434874
### Symptom 20: Pace Agent Sync Stalled by Corfu DB Distributed Streaming Exception UI Manifestation: Full sync operations delay indefinitely. Describing the cluster site layout resource shows processing statuses stuck In Progress but without producing distinct stack traces or failure logs. How to Verify: 1. Log into the SSP Installer VM and review the running status block details: k describe site -n nsxi-platform 2. Log into the NSX Manager CLI and search for Corfu cluster distribution streaming timeouts: grep -i "StreamingException" /var/log/proton/nsxapi.log	KB 435813
### Symptom 21: Multi-Tiered Intermediate Certificates Chain Breakdown UI Manifestation: No valid license available banner. Onboarding status looks active but fails to gather operational environment attributes (version number, deployment sizes, and host health states return blank). How to Verify: Log into the SSP Installer VM and read the real-time site-service pod log lines to look for platform controller handshake exceptions: k logs deploy/site-service -n nsxi-platform | grep -A 3 "remote-ui-plugins" Example Log Match: Delete ".../remote-ui-plugins/pace-ui": x509: certificate signed by unknown authority	KB 419764
### Symptom 22: Stale Certificate Profile Sync Deadlock Post SSP Recovery/Restore UI Manifestation: After recovering or restoring a backup image onto an SSP instance where Platform CA/Ingress keys were rotated previously, manual reconnect fails and Infrastructure Sync: DOWN persists. How to Verify: Log into the SSP Installer VM and parse the complete output of your platform site deployment resource yaml for certificate production failures: k get sites -n nsxi-platform -o yaml | grep -A 2 "COMMON_FULLSYNC" Example Log Match: message: 'COMMON_FULLSYNC failed due to: java.lang.Exception: produceCertMsgs' reason: FullSyncNotReady	KB 415360

Additional Information

Post off-boarding the NSX Manager there is a known issue while re-onboarding it back.

Please refer to Modifications to private IP ranges are retained even after the NSX Manager has been off-boarded for more information.