Persistent VCF upgrade Tasks with "400 BAD_REQUEST: The component is already at version"
Article ID: 453246
Updated On:
Products
Issue/Introduction
- In VCF Operations, at Build > Lifecycle > VCF Management > Upgrade, one or more components remain listed as Ready for upgrade after their upgrades have already completed successfully.
- The Upgrade path column shows the same build on both sides, for example
9.1.0.0400.########→9.1.0.0400.########, rather than a lower source version and a higher target version. - Running prechecks against an affected component reports that the component is already at the requested version, while the component may also display Prechecks failed, full results.
- Clicking UPGRADE on an affected component does not clear the entry.
- Restarting the lifecycle services does not clear the entries.
- The condition spreads. A component that upgrades successfully afterwards can enter the same state instead of clearing, so the number of affected components increases over time.
- Components confirmed as correctly installed and running their target versions continue to appear as outstanding work, which blocks subsequent upgrades on the same plan.
Additional symptoms reported:
- The application shows upgraded but the task still resides in Lifecycle saying pending upgrade.
- When you run prechecks it says the component is already on that version.
- This is preventing other components from upgrading.
Environment
VCF Operations 9.1, managing a VCF Management fleet with multiple components deployed on the VCF services platform.
Cause
The lifecycle upgrade process records progress in two places. Component level execution records track each attempt against a component, and plan level execution records track the overall run.
When several component upgrades are started at the same time against the same upgrade plan, those runs interfere with one another. Some execution records are never brought to a terminal state and remain marked as failed or as still running, even though the components themselves finish upgrading successfully.
The lifecycle interface reads these execution records to determine what work remains outstanding. Because the records never close, completed upgrades continue to be presented as available work. The version the record requests matches the version already installed, which is why the upgrade path shows the same build on both sides. No action available through the interface clears the entry, because the interface has no mechanism to close an execution record that the upgrade process abandoned.
Affected components remain fully functional and correctly upgraded throughout; only the lifecycle records are inaccurate.
Resolution
Tentatively planned to be fixed in a future release. Subscribe to this article to receive updates on this issue.
Prevent the condition
Trigger component upgrades one at a time. Do not use UPGRADE (ALL), and do not start an upgrade for a second component while another is still running.
- In VCF Operations, go to Build > Lifecycle > VCF Management > Upgrade.
- Select a single component and click UPGRADE.
- Wait for that upgrade to reach a completed state before starting another. Confirm completion on the Tasks tab before continuing.
- Repeat for each remaining component in turn.
Running upgrades sequentially avoids the interference that strands the records, and is the recommended practice on VCF Operations 9.1 until this issue is resolved.
If components are already affected
Entries that are already stuck cannot be cleared from the interface, and no customer-executable workaround is available. Correcting them requires changes that must be performed by Broadcom Support.
- Open a case with Broadcom Support and reference this article.
- Collect an SDDC Manager and VCF services platform log bundle, following Collecting SDDC Manager and VMSP logs, and upload it to the case using Uploading files to cases.
- Capture a screenshot of Build > Lifecycle > VCF Management > Upgrade showing the affected components and their upgrade paths, and attach it to the case.
- Take a snapshot and a backup of the SDDC Manager appliance. Support will ask for a current rollback point before making any change.
- Expect the correction to be performed during a scheduled live session. Support arranges engineering attendance as part of that session.
Do not attempt to modify the lifecycle database directly. Doing so without engineering guidance risks the consistency of the upgrade plan across the entire fleet.
Additional Information
Distinguishing this issue from KB 445069
Error: Undefined upgrade task in VCF 9.1 Operations lifecycle interface describes similar stale upgrade entries in the same interface, but a different underlying condition with a different resolution. The script attached to that article does not correct the issue described here.
Use these signals to determine which article applies to your environment:
| What you observe | KB 445069 | This article |
|---|---|---|
| Entry shown in the Upgrade list | An undefined entry, or an error reading Internal Server Error for service fleet-upgrade-service | A real component name with the same build shown as both source and target |
| Required binaries check | May report Unable to retrieve information about required binaries | Reports normally |
| What preceded the entries appearing | A failed installation or upgrade cleanup | Several component upgrades started at the same time |
| Concurrent upgrade activity | None active at the time of assessment | Concurrent upgrades are the trigger |
| Result of the KB 445069 script | Removes the stale entries | Reports No rows with eligibility_status='INELIGIBLE' were found. Nothing to clean up. and changes nothing |
If you have already run the cleanup_upgrade_plan_component.py script from KB 445069 and it reported that nothing needed cleaning up while your entries remained, this article describes your condition. Running that script causes no harm when no matching rows exist, and its output is useful to attach to your support case.
The two conditions differ in where the inaccurate records sit. KB 445069 addresses the component records the script is written to clean. This issue leaves the upgrade execution records incomplete, which that script does not examine.
Impact
Components affected by this issue are correctly installed and operating normally. The inaccuracy is confined to the lifecycle records, so there is no impact to workloads running on the affected components.
Feedback
