Error: Undefined upgrade task in VCF 9.1 Operations lifecycle interface

Products

• VMware SDDC Manager / VCF Installer
• VCF Operations

Issue/Introduction

Stale upgrade plan components appear in the VCF Operations lifecycle 9.1 interface. as Log Management lifecycle interface after a failed VCF 9.1 installation or upgrade cleanup.

• Components typically show a status:
  "Ready for upgrade Prechecks failed" and target versions such as 9.1.0.0.########. Attempting to re-run prechecks or resume the upgrade results in a VCF_LCM_FLEET_UPGRADE_400_INVALID_UPGRADE_PLAN_SPEC error.
• After running the cleanup_component script attached to Failed component cleanup during VCF 9.1 installation or upgrade, an "Internal Server Error" related to binaries may be present:
  Unable to retrieve information about required binaries. Internal Server Error for service fleet-upgrade-service. Verify logs for service fleet-upgrade-service.
  
• An undefined/failed upgrade entry remains visible as well after successful VCF Automation upgrade.
  
• Environmental assessment confirmed no active concurrent upgrade processes or precheck tasks.
• Selecting the stale record checkbox in the VCF Operations 9.1 UI and attempting to Run Precheck or Reconfigure fails with the following error:

  Failed to run action on component. Upgrade plan validation failed. The following validation errors were detected: VCF_LCM_FLEET_UPGRADE_400_INVALID_UPGRADE_PLAN_SPEC. Review the validation errors X and correct the input data. Ensure all component IDs in the filter exist in the upgrade plan inventory. Verify that all required fields are provided and have valid values.

  

• The respective task logs from the vcf fleet build pod (kubectl logs -n vcf-sddc-lcm vcf-sddc-build-service-sddcbuild-###############) show entries similar to:
  

  ERROR sddc-build-service <appi and logger details>
  Failed to authenticate OkHttp request GET https://<VCFops-fqdn>/api/vl/components/<component-id>: <component-fqdn>: Name or service not known
  ERROR sddc-build-service <appi and logger details>
  Failed to authenticate OkHttp request GET https://<VCFops-fqdn>/api/vl/components/<component-id>: <component-fqdn>: Name or service not known
  
  ERROR sddc-build-service <API and logger details>- Request failed after 6ms: GET https://<VCFops-fqdn>/api/vl/components/<component-id>: <component-fqdn>:
  java.net.UnknownHostException: <component-fqdn>: Name or service not known
  at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr (Native Method)
  at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr (Unknown Source)
  at java.base/java.net.InetAddress$PlatformResolver.lookupByName (Unknown Source)
  at java.base/java.net.InetAddress.getAddressesFromNameService (Unknown Source)
  ...
  Caused by: java.net.UnknownHostException: <component-fqdn>: Name or service not known

Environment

VCF Operations 9.1
VCF Automation 9.1

Cause

A stale upgrade_plan_component ID remains in the Fleet Lifecycle database (vcffleetlcmdb) after the cleanup_component.py script is executed. The UI renders these orphaned records as undefined tasks because the primary inventory references have been removed, but the task metadata persists.

Resolution

1. Login into VCF Operations and go to  Lifecycle > Build > VCF Management > Component > VCF services runtime 

   

2. Identify a Control Plane node IP:
   
   

3. Copy the attached script to the SDDC Manager /home/vcf/ directory.

4. SSH into the SDDC Manager with the vcf user and then elevate to root with su.
   
   
5. Run the script 
   
   python cleanup_upgrade_plan_component.py

Provide the VSP services runtime Control Plane node IP from step 2 when prompted.
Provide the vmware-system-user password when prompted. 
Follow the script instructions to identify and cleanup the stale upgrade_plan_component id.
   
   
6. Refresh the VCF Operations UI to confirm the failed upgrade task has been removed.

Additional Information

Failed component cleanup during VCF 9.1 installation or upgrade