NSX Upgrade Fails Precheck Due to Traceflow Data Inconsistency Deadlock

Products

• VMware NSX

Issue/Introduction

- During an automated NSX upgrade (via SDDC manager), the Management Plane (MP) precheck (subtask-2) fails. The worker log reports data inconsistencies across the NSX Manager nodes. Following is one of the example with error snippet:

"NSX prechecks failed for the component MP with error(s) : NSX-Mgr0/1/2 : Found data inconsistencies: Found 3 stale records in Traceflow Config, 72 stale records in Traceflow Observation, and 3 stale records in Traceflow Status tables."

The NSX UI (Plan & Troubleshoot -> Traffic Analysis -> Traceflow) does not display these stale traceflow records, indicating a discrepancy between the backend database and the user interface.

- Querying the pre-upgrade check failures API (/api/v1/upgrade/pre-upgrade-checks/failures) confirms the presence of stale traceflow configurations and observations.

Environment

VMware NSX 4.1.2

Cause

Stale traceflow configurations and observations persist in the Corfu database, preventing the pre-upgrade validation from completing successfully.

Resolution

1. Log in to the NSX Manager appliance as root via SSH or console.

2. Identify the stale traceflow records by executing the following API GET request: curl -k -u "admin" -X GET "https://localhost/policy/api/v1/traceflows"

3. Review the JSON output and note the unique id for each traceflow returned in the results array.

4. Delete each stale traceflow record by executing the following API DELETE request, replacing <traceflow-id> with the IDs identified in the previous step: curl -k -u 'admin' -H "X-Allow-Overwrite: true" -X DELETE "https://localhost/policy/api/v1/traceflows/<traceflow-id>

5. Verify the deletion was successful by running the GET request again and confirming the result_count is 0: curl -k -u "admin" -X GET "https://localhost/policy/api/v1/traceflows"

6. Retry the NSX-T upgrade prechecks to proceed with the upgrade.

Additional Information

Found data inconsistencies in Traceflow Observation during VCF 9.1 Precheck upgrade

Traceflow Operations Failing with "Traceflow Request Timeout" in NSX