Log management, Salt master or other VSP component upgrade workflow fails however the component shows upgraded version

Products

• VCF Operations

Issue/Introduction

During an upgrade of VMware Cloud Foundation (VCF) Fleet to version 9.1.0.0400, the Log Management, Salt Master or any other VSP component looks updated but the update task is failed.

• The task "Perform Log management upgrade" fails at the vmsp_upgrade step in the SDDC Manager UI.

• Attempting to retry the failed task results in an immediate, repeated failure.

• The SDDC Manager / Lifecycle Management (LCM) logs show the following error:
  An unexpected error occurred in step vmsp_upgrade. Reference Code: 396F3D16. Please contact support with this reference code. Detail: Error while extracting response for type [com.broadcom.vcf.lcm.sddc.upgrade.sdk.client.vmsp.model.Task] and content type [application/json]

Environment

VCF Operations 9.1.x

Log management 9.1.x

Cause

This issue is caused by a combination of two backend software defects within the VMSP service preventing SDDC Manager from properly reading the task status:

1. False-Positive IP Pool Exhaustion: The VMSP IPAM plugin incorrectly evaluates how IP claims are bound to virtual machines, leading it to mistakenly report that the IP pool is exhausted (specifically citing unknown-pool), even when ample IP addresses are available.

2. Malformed Error Message Payload: When generating the false-positive error, the fallback message is generated with empty values for the mandatory Level and TimeStamp fields.

When the SDDC Manager LCM service polls the task to check its status, its JSON parser crashes upon encountering the empty Level value. Because it cannot read the status, it crashes before it can even evaluate if the task is retryable, resulting in a permanent failure loop.

Resolution

To resolve this issue, manually patch the malformed JSON message in the backend cluster so SDDC Manager can parse it and proceed with the retry logic.

Important: Do not attempt manual database cleanup (e.g., modifying upgrade_plan_component or running cleanup_component.py) for this specific issue, as fixing the JSON payload natively unblocks the UI retry mechanism.

Prerequisites: Verify IP Capacity Before proceeding, confirm this is actually a false-positive and that the cluster is not genuinely out of IP addresses:

• Login to VCF services runtime control plane node with vmware-system-user username and run below commands
  
  
  • sudo su
  • export KUBECONFIG=/etc/kubernetes/admin.conf
  • kubectl get inclusterippool -A
  • kubectl get inclusterippool <pool-name> -n vmsp-platform -o jsonpath='{.status.ipAddresses}'

 Ensure that there is free capacity. If the IP pool is genuinely exhausted, do not proceed with the steps below and expand the IP pool instead.

1. Backup the Task Custom Resource Find the exact Task ID from the failed task and back it up before making modifications.
   
   
   • Note: upgrade-component- is a required constant prefix for the <TASK_ID> (for example: upgrade-component-7xkcrij62zhd3pja2urrsfqyim). The unique ID string can be found in the first 'Perform Log management upgrade' subtask which failed in the UI.
     
     
     • kubectl get task <TASK_ID> -n vmsp-platform -o json
     • kubectl get task <TASK_ID> -n vmsp-platform -o json > task-backup.json
       
       
       
2. Identify the Corrupted Message Index

   (Replace the timestamp value from the step 2 output. This just needs to be a placeholder and does not need to be accurate. Ensure angle brackets < > are not used around the timestamp value).

    
   Check the task messages to find the exact array index containing the corrupted payload:
   
   • kubectl get task <TASK_ID> -n vmsp-platform -o jsonpath='{.status.messages}' | jq .


     • Look through the output for the entry mentioning unknown-pool with empty fields for "level": "" and "timestamp": "". Note the array index number for this entry. This will be referred to as <IDX>.
     • Important: The array index (<IDX>) is zero-based, meaning counting starts from 0. For example, if the corrupted payload is the 11th item in the array, the index (<IDX>) will be 10. Verify this by counting the 'args' arrays in the output above.
       
       
       
3. Patch the Task Status Using the <TASK_ID> and the <IDX> number identified in the previous step, patch the custom resource to inject valid placeholder values.
   
   
   • Using the <TASK_ID> and the <IDX> number just identified, patch the custom resource to inject valid placeholder values:
   • Replace the timestamp value from the step 2 output. This just needs to be a placeholder and does not need to be accurate. Ensure not use angle brackets < > around the timestamp value).
     
     
     • kubectl patch task <TASK_ID> -n vmsp-platform \
  --subresource=status --type=json \
  -p '[
    {"op":"replace","path":"/status/messages/<IDX>/level","value":"ERROR"},
    {"op":"replace","path":"/status/messages/<IDX>/timestamp","value":"2026-07-14T16:14:40Z"}
  ]'
       
       
       
4. Verify the Task is Retriable Check that the task is now eligible for a retry:
   • kubectl get task <TASK_ID> -n vmsp-platform -o jsonpath='{.status.retryable}'
     
     
     • If this returns true, proceed to Step 5.
     • If this returns false or the subsequent retry fails, please Contact Broadcom Support for further assistance, as a manual task-reset may be required by engineering.
       
       
       
5. Retry the Upgrade
   Return to the SDDC Manager UI and trigger a Retry on the failed vmsp_upgrade task. Because the JSON payload is now correctly formatted, SDDC Manager will successfully read the task status and bypass the parsing crash, allowing the original upgrade to complete.

Additional Information

If the task has been failed for more than 20 days, the following issue may also be encountered: Cosmetic Task Failure (404 Not Found) After Retrying a Delayed Upgrade in VCF Operations.