NSX upgrade pre-check or actual upgrade fails due to SegmentRelationshipsMigrationTask failure

Products

• VMware NSX

Issue/Introduction

• Upgrading to NSX 3.2.4 or 4.1.x.
• Pre-check fails with error on UI:
  NSX Manager upgrade dry run failed. Do not proceed with the upgrade.
Please collect the support bundle and contact VMWare GS. Failed migrations: 
SegmentRelationshipsMigrationTask: null Exception during Data migration
• Policy Segment object has connectivity_path set to empty.
  For example: "connectivity_path": ""
  
  • To list Policy Segment objects, using NSX API
    GET /policy/api/v1/infra/segments
    Using curl:
    curl -k -u admin https://<NSX_Manager_IP>/policy/api/v1/infra/segments


    {
  "type" : "DISCONNECTED",
  "subnets" : [ {
    "gateway_address" : "192.168.100.1/24",
    "network" : "192.168.100.0/24"
  } ],
  "connectivity_path" : "",
...

• Related Manager logs:
  • /var/log/syslog
    <TIMESTAMP> <hostname> NSX 17209 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=<null>,name=Upgrade Evaluation Tool execution check,description=Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade.,componentType=MP] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=<null>,name=Upgrade Evaluation Tool execution check,description=Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade.,componentType=MP], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":36000,"errorMessage":"NSX Manager upgrade dry run failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GS. Failed migrations: SegmentRelationshipsMigrationTask: null#012Exception during Data migration#012"}]}
<TIMESTAMP> <hostname> NSX 17209 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [PUC] Finish Upgrade Evaluation Tool execution check on component MP. Result -- FAILURE
  • /var/log/upgrade-coordinator/logical-migration.log
    <TIMESTAMP>  INFO pool-#-thread-# MigrationTask 21072 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Handling segment /infra/segments/<SEGMENT_ID>
<TIMESTAMP> ERROR pool-#-thread-# PolicyPathUtil 21072 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500012" level="ERROR" subcomp="manager"] Invalid path
<TIMESTAMP>  WARN pool-#-thread-# UfoCorfuTableMigrator 21072 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] ERROR while running logical migration MappingDetails{modelName='null', migrationType=null, reason='This migration task will add missing relationships in Segment intent that are missing but, path is consumed by Segment.', customMigratorClassName='com.vmware.nsx.management.migration.impl.SegmentRelationshipsMigrationTask', fieldMappings=null, targetProtoName='null', requiresCustomCode='false', owner='null', apiToTest='null'}
com.vmware.nsx.management.policy.policyframework.exceptions.InvalidPolicyPathException: null
        at com.vmware.nsx.management.policy.policyframework.service.PolicyPathUtil.getPolicyPath(PolicyPathUtil.java:361) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.SegmentRelationshipsMigrationTask.migrateSegments(SegmentRelationshipsMigrationTask.java:150) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.SegmentRelationshipsMigrationTask.migrate(SegmentRelationshipsMigrationTask.java:91) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.ufo.UfoCorfuTableMigrator.migrate(UfoCorfuTableMigrator.java:138) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.ufo.UFOMigration.invokeMigrate(UFOMigration.java:403) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.ufo.UFOMigration.migrateData(UFOMigration.java:352) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.ufo.UFOMigration$1.run(UFOMigration.java:245) ~[logical-migration.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_341]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_341]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_341]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_341]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_341]
<TIMESTAMP>  INFO pool-#-thread-# ProgressMonitoring 21072 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Marking task SegmentRelationshipsMigrationTask as failed
    
    

• In rare cases, actual upgrade fails for the same data migration failure.
  Invoke "get upgrade progress-status" on the orchestrator node and you see it stopped on SegmentRelationshipsMigrationTask.
  

  <Host name>> get upgrade progress-status
****************************************************************************
Node Upgrade has been started. Please do not make any changes, until
the upgrade operation is complete. Run "get upgrade progress-status"
to show the progress of last upgrade step.
****************************************************************************
<Timestamp>
Upgrade info:
From-version: <Old veresion>
To-version: <New veresion>

  Upgrade steps:
download_os [<Timestamp> - <Timestamp>] SUCCESS
shutdown_manager [<Timestamp> - <Timestamp>] SUCCESS
install_os [<Timestamp> - <Timestamp>] SUCCESS
migrate_manager_config [<Timestamp> - <Timestamp>] SUCCESS
switch_os [<Timestamp> - <Timestamp>] SUCCESS
reboot [<Timestamp> - <Timestamp>] SUCCESS
run_migration_tool [<Timestamp> - <Timestamp>] FAILED
------ Output of last step start ------
Status:
<Timestamp> Deleting datastore files
<Timestamp> Copying old datastore files
<Timestamp> Done copying old datastore files
<Timestamp> Start Corfu server
<Timestamp> Process corfu-server started
<Timestamp> Error running logical data migration tool. return value 1, log file /var/log/proton/logical-migration.log

  Overall Progress: (3/6)
---- (1) CCP: Completed [1 object(s)] (<Timestamp> - <Timestamp>) ----
--------------------------------------------------------------------------------------------
---- (2) Proton: Completed [22435 object(s)] (<Timestamp> - <Timestamp>) ----
--------------------------------------------------------------------------------------------
---- (3) Policy: Completed [25078 object(s)] (<Timestamp> - <Timestamp>) ----
--------------------------------------------------------------------------------------------
---- (4) Logical: 47% [11432 of 24216 object(s)] (<Timestamp> - ) ----
Currently Migrating: SegmentRelationshipsMigrationTask 0% [0 of 189 objects] (<Timestamp> - )
--------------------------------------------------------------------------------------------
---- (5) CBM: Pending
--------------------------------------------------------------------------------------------
---- (6) UFO Checkpointing: Pending
--------------------------------------------------------------------------------------------
Stdout: Starting Manager run_migration_tool script
Ending Manager run_migration_tool script

  Troubleshooting: Upgrade has failed, and retry may not work. Appliance OS is of a new version, however, UI will not be available. Please contact GSS to rollback the system to the previous version.
------ Output of last step end ------

Environment

NSX-T Data Center 3.2.4, 3.2.4.1

NSX 4.1

Cause

Pre-check cannot process Segment with connectivity_path set to empty ("") and mark the segment migration task as failed.

Usually it is seen on precheck, however, if you create or edit a segment after you trigger upgrade, the actual upgrade might fail and all the NSX Managers might be in failed state.
Please refrain from making any change after you trigger upgrade.

Resolution

Upgrade to 4.2 or later instead of 3.2.4 or 4.1.x.

Alternatively, removing a line '"connectivity_path": ""' from each Policy Segment objects resolves pre-check failure.

1. Confirm backup is completed.
2. List Policy Segment objects, using NSX API.
   GET /policy/api/v1/infra/segments
   
   Using curl to list Segment objects:
   curl -k -u admin https://<NSX_Manager_IP>/policy/api/v1/infra/segments/
   
   
3. Find ID field where connectivity_path is set to empty (""). This ID is Segment ID.
4. Get Policy Segment object with Segment ID.
   GET /policy/api/v1/infra/segments/<segment_id>
   
   Using curl to save the response to a file "Segment.json"
   curl -k -u admin https://<NSX_Manager_IP>/policy/api/v1/infra/segments/<segment_id> > Segment.json
   Replace segment_id with Segment ID.
   
   
5. Remove the line '"connectivity_path": ""' from the response.
6. Update with the edited response without connectivity_path.
   PUT /policy/api/v1/infra/segments/<segment_id>
   
   Using curl:
   curl -k -u admin [e-mail] -X PUT -H'Content-type: application/json' https://<NSX_Manager_IP>/policy/api/v1/infra/segments/<segment_id>
   Replace segment_id with Segment ID.
   In this example, the fixed request body which does not contain connectivity_path is saved to the file "Segment.json".

On GUI, connect the segments to a gateway and then remove the connection.

After upgrade precheck succeeds, you can proceed upgrade.
Also refrain from making any change after you trigger upgrade.

If an actual upgrade fails, contact Broadcom support.