Handling LCM error "UPGRADE_SPEC_INVALID_DATA" during vLCM upgrade scheduling

Products

• VMware Cloud Foundation

Issue/Introduction

Users should be able to trigger vLCM cluster upgrades via API in their upgrade workflow without error.

Symptoms:
A user can have an environment with mixed cluster types, both VUM and VLCM.
During an upgrade window, if a user has attempted to schedule parallel ESXi upgrades and they are in progress, then tries to schedule vLCM upgrades in draft mode, upgrades API can return: 

{
    "errorCode": "UPGRADE_SPEC_INVALID_DATA",
    "arguments": [],
    "referenceToken": "BCODID"
}.

With an associated stack trace in the LCM error logs:

com.vmware.evo.sddc.common.core.error.InvalidInputException: Upgrade Spec Invalid Data: Upgrade pending on vcenter
<vcenter-id>.

Error in  /var/log/vmware/vcf/lcm/lcm-debug.log 

2023-04-29T13:53:08.480+0000 ERROR [vcf_lcm,d5604274bad1429c,6f59] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7400-exec-92] [BCODID] UPGRADE_SPEC_INVALID_DATA Upgrade Spec Invalid Data: Upgrade pending on vcenter 94bcccd7-695e-4bda-82d9-e162b5855e58.

Environment

VMware Cloud Foundation 5.0

Cause

The issue occurs when the user has already triggered vLCM cluster upgrades and moves to draft mode.
LCM considers any draft mode as VMware upgrade pending and throws an error while attempting to upgrade for the same resource.

Resolution

This issue is resolved in VMware Cloud Foundation 5.1.

Workaround:
Follow the below steps:

Step 1: Confirm that there are upgrades in draft mode by checking the database.

• ssh to sddc manager with vcf and su to root
• Run the command - psql -h localhost -U postgres -d lcm -c "select upgrade_id,upgrade_status from upgrade where upgrade_status = 'DRAFT'"

Step 2: If there are values populated by this, then you can cancel the vLCM upgrades in the UI, after which the draft mode upgrades should go away in the table. You can view which upgrades are in draft mode via following the same psql commands as Step 1.

Step 3: If upgrades still persist in the database, they can remove the DRAFT status.

• Take snapshot of SDDC Manager VM
• ssh to sddc manager with vcf and su to root
• Run the command - psql -h localhost -U postgres -d lcm -c "DELETE FROM upgrade WHERE upgrade_status='DRAFT'"