Found data inconsistencies in Traceflow Observation during VCF 9.1 Precheck upgrade

Products

VMware NSX VMware Cloud Foundation VCF Operations

Issue/Introduction

Upgrade Pre-check for NSX performed from SDDC end.
VCF upgrade to 9.1 from prior to 9.1 versions
Pre-check error on the SDDC UI

No running traceflow seen from API GET https://<mgr-ip>/policy/api/v1/infra/traceflows
Rolling reboot of the NSX Managers did not clear the pre-check error.
Below logs are seen on the NSX manager node where service install-upgrade was held.

upgrade-coordinator.log

[TIMESTAMP] INFO pool-36-thread-2 UpgradeServiceImpl 4057624 SYSTEM [nsx@4413 comp="nsx-manager" level="INFO" logger="UpgradeServiceImpl" ms
gID="SYSTEM" subcomp="upgrade-coordinator" threadName="pool-36-thread-2"] [PUC] Finish Check for data inconsistencies in DB on component MP. Result -- Bas
icInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[acknowledgement=false,componentType=MP,description=This precheck aims to identify and r
esolve data inconsistencies in the NSX Manager database.,id=dataConsistencyCheckCheck-StaleTraceflowDetectorTask,name=Check for data inconsistencies in DB
,needsAcknowledgement=false,needsResolution=true,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordina
tor","errorCode":31911,"errorMessage":"Found data inconsistencies: Found 0 stale records in Traceflow Config, 460 stale records in Traceflow Observation,
and 7 stale records in Traceflow Status tables. Traceflow Observation IDs: [/infra/traceflow-observations/########-####-####-####-############,....]

When checking on the below NSX corfu table, we could see the entries.
/opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t TraceflowObservation
/opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t TraceflowStatus

Environment

VMware NSX 9.1

VMware Cloud Foundation 9.1

Cause

This precheck error is for removing the remaining stale resources in the system.

Resolution

Follow any one of the below options to resolve the error:

(Option: A)

Step 1 : Confirm the failing check ID and needs_resolve=true: using below API command (from root mode of NSX Manager CLI)

curl -k -u 'admin:<PASSWORD>' "https://<NSX-MANAGER-VIP>/api/v1/upgrade/pre-upgrade-checks/failures" -H "Accept: application/json"

Expected response excerpt:

{
  "results": [
    {
      "type": "FAILURE",
      "message": {
        "message": "Found data inconsistencies: Found 0 stale records in Traceflow Config, 460 stale records in Traceflow Observation, and 7 stale records in Traceflow Status tables.",
        "error_code": 31911
      },
      "component_type": "MP",
      "origin_type": "COMPONENT",
      "needs_resolve": true,
      "resolution_status": "UNRESOLVED",
      "id": "dataConsistencyCheckCheck-StaleTraceflowDetectorTask"
    }
  ]
}

Step 2 : Invoke the resolve

curl -k -u 'admin:<PASSWORD>' -X POST "https://<NSX-MANAGER-VIP>/api/v1/upgrade/pre-upgrade-checks/dataConsistencyCheckCheck-StaleTraceflowDetectorTask?action=resolve" -H "Content-Type: application/json"

This calls deletes all TraceflowObservation and TraceflowStatus stale Corfu records.

Step 3 — Verify resolve succeeded:

curl -k -u 'admin:<PASSWORD>' "https://<NSX-MANAGER-VIP>/api/v1/upgrade/pre-upgrade-checks/failures" -H "Accept: application/json"

The StaleTraceflowDetectorTask failure should now have resolution_status: "RESOLVED" (or disappear entirely after a fresh precheck run).

(Option: B)

Use the same .pub file having same NSX version.
Upload on the node where the install upgrade service resides from NSX UI
Run pre-checks and Click on "Resolve" for the Traceflow error.
Run same precheck again on NSX and it should pass.
Run the precheck upgrade from SDDC, it should all pass.

For any assistance, contact Broadcom Support