vCenter upgrade option grayed out in SDDC Manager with DOMAIN_HAVE_DRAFT

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

In VMware Cloud Foundation (VCF), the vCenter upgrade option is grayed out and unavailable within the SDDC Manager UI.

After upgrading the NSX Manager in VCF 9.1, the vCenter upgrade option has become grayed out.

vCenter upgrade Tab is not getting highlighted. The KB 444168 was validated, however, we can see the correct target Version of 9.1 when we run a GET /v1/releases/domains

When we attempt to use the "Plan Domain Upgrade" feature and click on "Target Version", we see two alerts:

Info alert: To activate or deactivate compatibility checks on the target version, see KB 90074
When we select the VCF version, all the versions shows an error.
- It complains about: You cannot select this target version for the following reasons. Select a different target version to proceed.
  Domain <Domain ongoing upgrade> has update in draft state.

Users may encounter the error "Saving domain target version failed" when attempting to upgrade vCenter or save a workload domain target version.

Associated error messages often include:

"Validation failures occurred while trying to save target version"
"Reference token has update in draft state"

Review of the /var/log/vmware/vcf/lcm/lcm-debug.log confirms the workload domain is blocked from calculating applicable upgrades due to an update stuck in a draft state, presenting the following errors:

YYYY-MM-DDTHH:mm:ss.sss+zzzz ERROR [vcf_lcm,0d163a57f6214509,b647] [c.v.e.s.l.s.v.u.ValidationHelperUtil,http-nio-127.0.0.1-7400-exec-10] Domain ###### has update in DRAFT state.
DomainFutureRelease(super=Release(product=VCF, version=9.0.0.0), applicabilityStatus=NOT_APPLICABLE, notApplicableReason=LocalizableMessagePack(component=VCF, messageKey=DOMAIN_HAVE_DRAFT_UPDATE, arguments=[#####]...

Environment

VMware Cloud Foundation 9.1.x

SDDC Manager 9.1.x

Cause

An incomplete or abandoned update plan was left in a pending "DRAFT" state within the LCM database. This active draft locks the workload domain, preventing SDDC Manager from allowing new patching or upgrade plans to be initiated.

Resolution

Steps for Clearing incomplete Draft Upgrade Plan

Note: Manually modifying the SDDC Manager lcm PostgreSQL database is an irreversible action. You must secure a valid snapshot of the SDDC Manager appliance before executing the database deletion step.

Take a snapshot of the SDDC Manager appliance.
Log in to the SDDC Manager UI.
Navigate to the upgrade section and attempt to cancel the Plan Domain Upgrade plan. If the UI cancellation fails with error: Cancel upgrade plan failed. Validation failures occurred while trying to delete target version Reference token: #####. (Proceed with the steps below to remove it from the database.)
Access the Database: SSH into the SDDC Manager appliance as the vcf user, then switch to the root user: su -.
Query the LCM database to confirm the draft state exists by executing the following command: (Attached Example Image Below)
- psql -h localhost -U postgres -d lcm -c "select upgrade_id,upgrade_status from upgrade where upgrade_status = 'DRAFT'"
  
  For example, the output should match the below:
  
  root@vcf [ ~ ]# psql -h localhost -U postgres -d lcm -c "select upgrade_id,upgrade_status from upgrade where upgrade_status = 'DRAFT'"
  WARNING: database "lcm" has a collation version mismatch
  DETAIL: The database was created using collation version 2.32, but the operating system provides version 2.36.
  HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE lcm REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
  upgrade_id | upgrade_status
  ---------------------------------------------------------+-----------------------
  <id> | DRAFT
  (1 row)
Manually remove the draft lock from the database by executing the following command:
- psql -h localhost -U postgres -d lcm -c "DELETE FROM upgrade WHERE upgrade_status='DRAFT'"
  
  For example, the output should match the below:
  
  root@vcf [ ~ ]# psql -h localhost -U postgres -d lcm -c "DELETE FROM upgrade WHERE upgrade_status='DRAFT'"
  WARNING: database "lcm" has a collation version mismatch
  DETAIL: The database was created using collation version 2.32, but the operating system provides version 2.36.
  HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE lcm REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
  DELETE 1
Restart the LCM service to recalculate future release applicability by executing the following command:
- systemctl restart lcm

Additional Information

Management Domain Upgrade Option for vCenter and NSX Greyed Out in VCF 9.1