SDDC Manager upgrade fails at CONFIG DRIFT with error "ADD_EDGE_NSXT_SOURCE_ID_POST_VALIDATE_FAILED"

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

•  The Config Drift bundle upgrade fails with error: ADD_EDGE_NSXT_SOURCE_ID_POST_VALIDATE_FAILED, checking sddcmanager_migration_app_upgrade.log:

Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to post-validate adding NSX-T source IDs to VCF edge inventory for edge cluster <Edge_Cluster_FQDN>

Failed to set source ID for edge cluster <Edge_Cluster_FQDN>, node <Edge_TN_FQDN>. Expected 98ea0ace-####-####-####-########c4f but found 6f738f03-####-####-####-########c52.

• Reviewing the postgres DB shows the old source ID for the NSX edge nodes:

psql -h localhost -U postgres -d platform
select * from nsxt_edge_cluster where name='<Edge_Cluster_FQDN>';

id               | 6aff6b70-####-####-####-########ece
creation_time    | 1624055551383 
modification_time| 1624055551383 
status           | ACTIVE 
name             | <Edge_Cluster_FQDN> 
nsxt_edge_nodes  | [{"vmManagementIpAddress":"<IP>","vmHostname":"Edge_TN_FQDN","sourceId":"6f738f03-####-####-####-########c52","id":"28bb2cc0-####-####-####-########3d3"}, {"vmManagementIpAddress":"<IP>","vmHostname":"Edge_TN_FQDN","sourceId":"ac7384a0-####-####-####-########36a","id":"b4203702-####-####-####-########259"}] 
source_id        | 554ba1f6-####-####-####-########080

• Similarly, the NSX-T postgres table displays the new ID for the edge nodes. 

"edgeClusterMembersTransportNodeIds": [
 "98ea0ace-####-####-####-########c4f",
"33c3bdef-####-####-####-########24c"
]

Environment

VMware Cloud Foundation 4.x

Cause

• This issue is commonly seen in VCF environments where edge nodes are redeployed from NSX Manager for resizing purposes i.e. changing edge from medium to large. 
• When redeploying NSX edges from NSX Manager directly, NSX Manager assigns a new ID for the redeployed edge node which will differ from the ID saved in SDDC DB causing the failure for SDDC CONFIG DRIFT upgrade.
• VCF environments should consider utilizing kb https://kb.vmware.com/s/article/87542 for resizing edge nodes.

Resolution

1. Take a snapshot of the SDDC manager VM.
2. Connect to SDDC using vcf then switch to root.
3. Run the following command:

curl localhost/inventory/nsxt-edgeclusters | json_pp > res.json

4. Search for the Edge cluster inside the file 'res.json' then replace the value written for the edge node in 'edgeNodeNsxtId' with the new NSXT ID seen in NSXT UI or error message.
5. Create a new file 'req.json' and put only your edge cluster payload with the updated 'edgeClusterNsxtId'.  Please make sure of removing the [] - the PUT request takes a single object and not an array in the request body.

Example req.json:

{
"clusterIds": [
"ba24f64e-####-####-####-########144"
],
"nsxtClusterId": "2f3d2746-####-####-####-########569",
"id": "6aff6b70-####-####-####-########ece", <---------------------- Edge clusterID in SDDC DB - We will need it to run our PUT API
"edgeClusterNsxtId": "554ba1f6-####-####-####-########080",
"name": "<Edge_Cluster_FQDN>",
"status": "ACTIVE",
"nsxtEdgeNodes": [{
"id": "28bb2cc0-####-####-####-########3d3",
"hostName": "<Edge_TN_FQDN>",
"managementIpAddress": "<IP>",
"edgeNodeNsxtId": "98ea0ace-####-####-####-########c4f" <---------------------- New edge transport node ID 
},
{
"id": "b4203702-####-####-####-########259",
"hostName": "<Edge_TN_FQDN>",
"managementIpAddress": "<IP>",
"edgeNodeNsxtId": "33c3bdef-####-####-####-########24e" <---------------------- New edge transport node ID 
}
]
}

  

6. Run  the command:

 curl localhost/inventory/nsxt-edgeclusters/{id} -v -X PUT -H 'content-type:application/json' -d @req.json

7. Retry upgrade.