4.2.3.x to 9.0.x Rolling Upgrade of NSX Manager may lead to 'Not Ready' status in SSP with Config Agent in backoff retry state during NSX-T upgrade
Article ID: 447957
Updated On:
Products
Issue/Introduction
During a rolling upgrade of NSX-T from version 4.2.3.x to 9.0.x, the Config agent component which is responsible on the NSX Manager enters an exponential backoff retry error state and is unable sync configs to SSP.
Symptoms:
SSP CLI (Primary Verification): Run the following command as sysadmin on the SSPI appliance to determine the failure state.
kubectl describe site -n nsxi-platform
For output indicating NsxConfigTOIUpdateFailed please contact GSS team for assistance. Do not proceed with next steps.
logs:
...
lastTransitionTime: "2026-07-15T11:49:41Z"
message: 'error while executing API call to https://nsx-config/api/v1/intelligence/config/sites {"error_code":900069,"module_name":"NsxIntelligence","error_message":"NSX TOI Update error: Internal server error has occurred."}'
reason: NsxConfigTOIUpdateFailed
type: NsxStreamingReady
..
For output indicating 'PACE_AGENT_INIT failed' confirms the known issue described in this article. Continue verifying the remaining symptoms below.
logs
...
lastTransitionTime: "2026-07-15T11:49:41Z"
message: 'error while executing API call to https://nsx-config/api/v1/intelligence/config/sites {"error_code":900069,"module_name":"NsxIntelligence","error_message":"NSX TOI Update error: Internal server error has occurred."}'
reason: NsxConfigTOIUpdateFailed
type: NsxStreamingReady
..
SSP UI: NSX Manager status may show Not Ready in the SSP UI.
NSX Manager CLI: Checking the proton logs on the NSX Manager shows a missing stream tag error
# Login to NSX manager
ssh root@<nsx-manager>
# Verify the logs
cd /var/log/proton
zgrep "does not have specified stream tag: intelligence" -A 3 nsxapi.*
#Outputs
java.lang.IllegalArgumentException: Interested table: nsx$ComputeCollectionMsg does not have specified stream tag: intelligence
Environment
- NSX source version 4.2.3.x
- NSX target version 9.0.x
- SSP 5.x
Cause
This issue is triggered because certain CorfuDB tables (e.g., `nsx$ComputeCollectionMsg`) fail to stream updates to the SSP due to missing stream tags. In a rolling upgrade, NSX Managers are upgraded to 9.0.x before Edges, Hosts, and vCenter. During this transitional phase, the upgraded Manager run 9.0.x code, but the overall platform version is still reported as 4.2.3.x in SSP. NSX versions 4.2.2 and later include IDPS (Intrusion Detection and Prevention System) features. These features required certain CorfuDB tables — such as nsx$ComputeCollectionMsg — to have streaming capability so that SSP could subscribe to updates from them. In the 9.0.x train, the IDPS-related features are not present. As a result, the streaming capability is missing for some tables resulting into an intermediate backoff retry state.
Note: This is a temporary intermediate state. The issue will automatically resolve itself once the rolling upgrade is fully completed across all components.
Resolution
The following tables should be removed from the request below:
ComputeCollectionMsgDiscoveredNodeMsgPolicyConfigComputeManagerModelMsgHostDiscoveredNodeRelationshipMsg
Step 1: Get the current consumer config and save the full response — use it as the base for the PUT request body in Step 2.
Note: The following REST APIs that can be run from any machine that has network access to NSX Manager.
If you get an SSL certificate error (self-signed cert), add the-kflag to skip verification:curl -k --request GET...Get Consumer Config:
curl --request GET \
--url https://<nsx-manager>/policy/api/v1/config-streaming-agent/consumers/SSP \
--user "<username>:<password>"
#Sample response
{
"consumer_name": "SSP",
"version": "SSP-TABLES-GT-44-CT-22-Metrics-Intelligence-MPS-RuleAnalysis-4.2.3.2.0.25077145",
"streaming_type": "KAFKA",
"groups": [
{
"streaming_address": "nsx2pace-config-group2",
"tables": [
{
"name": "ComputeCollectionMsg",
"versions": [
"GENERIC"
]
},
{
"name": "DiscoveredNodeMsg",
"versions": [
"GENERIC"
]
},
{
"name": "ComputeManagerModelMsg",
"versions": [
"GENERIC"
]
},
{
"name": "PolicyConfig",
"versions": [
"GENERIC"
]
},
{
"name": "HostDiscoveredNodeRelationshipMsg",
"versions": [
"GENERIC"
]
},
{
"name": "ContainerClusterMsg",
"versions": [
"CLASSIC"
]
},
..
]
},
{
"streaming_address": "nsx2pace-config-group1",
"tables": [
{
"name": "Group",
"versions": [
"GENERIC",
"CLASSIC"
]
},
...
]
}
]
}
Step 2: Use the response from step 1 as request body to update consumer config with the affected tables removed.
Update Consumer Config:
# Update consumer config with ComputeCollectionMsg removed from request body.
curl --request PUT \
--url https://<nsx-manager>/policy/api/v1/config-streaming-agent/consumers/SSP \
--user "<username>:<password>" \
--header 'content-type: application/json' \
--data '{
"consumer_name": "SSP",
"version": "SSP-TABLES-GT-44-CT-22-Metrics-Intelligence-MPS-RuleAnalysis-4.2.3.2.0.25077145",
"streaming_type": "KAFKA",
"groups": [
{
"streaming_address": "nsx2pace-config-group2",
"tables": [
{
"name": "ContainerClusterMsg",
"versions": [
"CLASSIC"
]
},
..
]
},
{
"streaming_address": "nsx2pace-config-group1",
"tables": [
{
"name": "Group",
"versions": [
"GENERIC",
"CLASSIC"
]
},
...
]
}
]
}'
Feedback
