VCF upgrade planning fails: Domain components are not in active state

Products

• VCF Operations

Issue/Introduction

While running the upgrade planning for VMware Cloud Foundation (VCF) components, such as updating a workload domain's vCenter via VCF Operations, the validation process fails to save the target version.

• The UI displays a validation error: Saving domain target version failed. Validation failures occurred while trying to save target version Reference token: QT###8: Domain example.com or some of its components are not in active state
  
  

• The LCM logs on the SDDC Manager (/var/log/vmware/sddc-support/lcm/lcm-debug.log) contain a localization error confirming the validation failure:

  ERROR [vcf_lcm,58b8d###431541b7,bd44] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7400-exec-2] [9###CF] LCM_SAVE_TARGET_VERSION_VALIDATION Validation failures occurred while trying to save target version com.vmware.evo.sddc.common.core.error.CompositeInvalidInputException: Validation failures occurred while trying to save target version
ERROR [vcf_lcm,58b8d###431541b7,bd44] [c.v.v.i.b.i.TranslationMessage,http-nio-127.0.0.1-7400-exec-2] Can't find resource for bundle java.util.PropertyResourceBundle, key LCM_SAVE_TARGET_VERSION_VALIDATION.remedy

Environment

VMware Cloud Foundation 9.x

Cause

This error occurs due to a stale inventory state in the SDDC Manager platform database. Automated tasks, such as PowerCLI-driven password rotations or host management operations, fail to complete, leaving hosts or domain entities in an ERROR status within the database. The ActiveStateDomainValidator requires all components to be in an ACTIVE state to proceed with the upgrade planning, and the localization error is a secondary symptom of the validator failing to map the specific remedy text for this state.

Resolution

1. Identify the components in the SDDC Manager which are in an ERROR state.

   1. SSH to the SDDC Manager appliance using the vcf user and su to root.

   2. Run the VCF Diagnostic Tool (VDT) to identify the component in an ERROR state. Reference: Using the VCF Diagnostic Tool for SDDC Manager.
      
      Log output example:
      
      INVENTORY STATUS
[PASS] Host Status Check
       All hosts are in an ACTIVE state.
[PASS] Domain Status Check
       All domains are in an ACTIVE state.
[FAIL] vCenter Status Check
       ########-####-####-####-#######4ce2 | vcenter.example.com | ERROR [PASS] PSCs Status Check All PSCs are in an ACTIVE state. [PASS] Cluster Status Check All Clusters are in an ACTIVE state. [PASS] NSX Manager Status Check All NSX Managers are in an ACTIVE state.

2. Update the component status to ACTIVE in the SDDC platform database.

   1. Create a snapshot of the SDDC Manager virtual machine.

   2. Establish an SSH connection to the SDDC Manager appliance as the vcf user, then elevate to root privileges using the su command.

   3. Connect to the platform database for VCF 9: psql -h localhost -U postgres -d platform.

   4. Update the component status to ACTIVE as identified in Step 1(b): update <database_table_name> set status='ACTIVE' where id='<id from step 1(b)>';
      
      Sample (Host): update host set status='ACTIVE' where id='########-####-####-####-########a1b2';
      
      Sample (vCenter): update vcenter set status='ACTIVE' where id='########-####-####-####-########4ce2';
      
      Sample (NSX): update nsxt set status='ACTIVE' where id='########-####-####-####-########e92d';

3. Retry the plan upgrade from the SDDC Manager UI.

4. (Optional) You may need to restart the SDDC Manager services: /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Additional Information

Use the following commands to check the specific tables manually:

psql -h localhost -U postgres -d platform -c "\x" -c "select * from domain where status !='ACTIVE';"
psql -h localhost -U postgres -d platform -c "\x" -c "select * from host where status !='ACTIVE';"
psql -h localhost -U postgres -d platform -c "\x" -c "select * from vcenter where status !='ACTIVE';"
psql -h localhost -U postgres -d platform -c "\x" -c "select * from nsxt where status !='ACTIVE';"
psql -h localhost -U postgres -d platform -c "\x" -c "select * from cluster where status !='ACTIVE';"
psql -h localhost -U postgres -d platform -c "select id, lb_hostname, status from vrli;"
psql -h localhost -U postgres -d platform -c "select id, lb_hostname, status from vrops;"
psql -h localhost -U postgres -d platform -c "select id, vrslcm_node, status from vrslcm;"
psql -h localhost -U postgres -d platform -c "\x" -c "select id,vm_hostname,status from psc;"
psql -h localhost -U postgres -d platform -c "\x" -c "select id,cluster_fqdn,status from nsxt;"
psql -h localhost -U postgres -d platform -c "select * from sddc_manager_controller;"