An upgrade fails with 409 Conflict - task.conflict due to stalled/orphaned for VCF Operations for Networks 9.1.1

Products

• VCF Operations for Networks
• VCF Operations

Issue/Introduction

• The upgrade of VCF Operations for networks  from 9.1.0.0200.25517220 to 9.1.1.0.25662699 fails at the final "Perform Component Upgrade" step with an HTTP 409 task.conflict error.
  
  Error seen is as below:
  
  

• Task status: Failed
  Subtask: Perform Component Upgrade
  Subtask status: Failed
  Severity: Error
  Message: An unexpected error occurred in step networks_apply_upgrade. Reference Code: 2BB679AE.
  Please contact support with this reference code. Detail: 409 Conflict: "{"code":"80001","message":"id":"com.broadcom.vcf.netops.upgrade.task.conflict","defaultMessage\":\"Another task of same category running.","localizedMessage":"Another task of same category running.","args":{}},"timestamp":"YYYY-MM-DDT01:39:46.789Z"}"

Environment

• VCF Operations for Networks 9.1.1
• VCF Operation 9.1.1

Cause

A prior VCF Operations for Networks  upgrade (9.1.0 GA → 9.1.0 EP2) would get orphaned by Fleet LCM due to a corner case issue that requires two independent, low-probability conditions in Fleet LCM to align:

1. A transient infrastructure-level DB/DNS blip hitting SDDC LCM's Conductor at the exact moment it both re-queues the polling of a prior upgrade's final step and runs its own reconciliation safety net, causing both to fail together.
   
   
2. A subsequent VCF Operations for Networks  upgrade being attempted afterward while that orphaned, non-terminal task record still sits in VCF Operations for Network's store. If the DNS blip doesn't hit both paths simultaneously, Conductor's normal self-healing sweep would have caught and cleared the stale task on its own.

Above two essentially leads to exploiting an issue in VCF Operations for Networks that left a stale UPGRADE_APPLY task record stuck in IN_PROGRESS.

Resolution

To Work around this issue, perform below steps:

1. Get the task_id of the previous upgrade from Failed Task from Fleet Lifecycle GUI that left the stale marker.
   
   This task_id is seen under message section on the Perform Component Upgrade in Fleet Lifecycle Tasks.
   
   Refer to below screenshot:
   
   
   
   
2. Login to VCF Operations for Networks platform node1 by taking SSH/Putty session and login with username support
   
   
3. Switch to ubuntu user by executing command ub
   
   
4. Get a new VCF Operations for Networks auth token (creds are ones used for GUI login), by executing  below curl command:
   
   curl -sk -X POST https://api/ni/auth/token -H 'Content-Type: application/json' -d '{"username":"","password":"","domain":{"domain_type":"LOCAL"}}'
   
   Example response: 
   
   {"token":"<token_id>","expiry":1786701898581}

5. Save above token_id as this will be used in Step 6.

6. Trigger the GET status API  with task_id captured in Step1 by executing below curl command: 
   

   curl -sk https://api/ni/upgrade/tasks/ -H 'Authorization: NetworkInsight <token_id>'
   
   In above command replace the token_id with the actual token_id captured as in output of step4.

7. Retry upgrade for 9.1.0 EP2 > 9.1.1, in few minutes it should trigger the upgrade for the component VCF Operations for Network.

Additional Information

This is a known issue impacting VCF Operations for Networks 9.1.1 upgrade. Subscribe to this KB article to be updated if a fix version becomes available.