Scheduling update in SDDC Manager fails with red banner "Scheduling immediate update of bundle failed. Upgrade is requested for domain 47d####-####-####-####-########90ac, which is in failed state"

Products

• VMware Cloud Foundation

Issue/Introduction

• SDDC Manager UI reports the below error
  
  
  
• Error in /var/log/vmware/vcf/lcm/lcm-debug.log

  ERROR [vcf_lcm,#######,c915] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7400-exec-7] [#####] DOMAIN_ID_INVALID Could not find the cluster resource details for the domain id 47d#####-####-####-####-########90ac
  com.vmware.evo.sddc.common.core.error.InvalidInputException: Could not find the cluster resource details for the domain id 47d#####-####-####-####-########90ac
  
  DEBUG [vcf_lcm,########,e728] [c.v.e.s.l.a.i.i.InventoryClientImpl,http-nio-127.0.0.1-7400-exec-9] Failed Resources Map: {ESX_HOST:27c#####-####-####-####-########5be8=1745438688254}
  2025-04-23T20:05:19.421+0000 ERROR [vcf_lcm,########,e728] [c.v.v.l.r.a.c.v.u.UpgradeController,http-nio-127.0.0.1-7400-exec-9] Scheduling Upgrade Failed

   

• Host ID reported in the log is in error state in SDDC platform database

  psql -h localhost -U postgres -d platform -c "select id,hostname,status from host where id='27c#####-####-####-####-########5be8';
  
                    id                  |          hostname           |    status
  --------------------------------------+-----------------------------+----------------+--------
   27c#####-####-####-####-########5be8 | esxi01.example.com          |	ERROR
  (1 row)

Environment

VMware Cloud Foundation

Cause

The status of vx_manager/host/domain for the corresponding clusters is in an ERROR state in the SDDC DB. 

Resolution

1. Take a snapshot of the SDDC Manager VM.
2. SSH to SDDC Manager using vcf account and elevate to root with su.
3. Check the domain itself is in ACTIVE state by running the below query

   psql -h localhost -U postgres -d platform -c "select * from domain"

4. Verify with the domain id of the domain which we are upgrading is in ACTIVE state
5. Check the status of hosts and they should be ACTIVE too:

   psql -h localhost -U postgres -d platform -c "select id, hostname, status from host"

6. Run the below query to check if the vx_manager is in an "ERROR" state

   psql -h localhost -U postgres -d platform -c "select * from vx_manager;"

7. Record the IDs of the component that are in ERROR state which are part of those clusters that we want to upgrade as part of the vxrail/host upgrade
8. Update those component(s) which is in "ERROR" state by setting them to "ACTIVE", the below command can be used for the same

   For Domain: 
   psql -h localhost -U postgres -d platform -c "update domain set status='ACTIVE' where id='<ids obtained from step 3>';"
   
   For Host: 
   psql -h localhost -U postgres -d platform -c "update host set status='ACTIVE' where id='<ids obtained from step 5>';"
   
   For vxRail: 
   psql -h localhost -U postgres -d platform -c "update vx_manager set status='ACTIVE' where id='<ids obtained from step 6>';"

9. Once all the components are updated to an ACTIVE state, cycle the SDDC services.

   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

10. Re-try the upgrade again