Error: "Domain ####-####-####-####-############ inventory state is not active" during SDDC Manager Plan upgrade

Products

• VMware SDDC Manager / VCF Installer
• VMware vCenter Server
• VMware vSphere ESXi
• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

• UPDATE NOW button grayed out with error:

  New update cannot be started while domain or some of its components are not in an active state
  

  
  
  
  
  
• Shared domain #### or some of its components are not in active state.
  
  

 

Environment

• SDDC Manager 4.x, 5.x, 9.x
• VCF Ops: 9.x

Cause

One or more components in the SDDC Manager database are in an ERROR or in a state other than ACTIVE state. 

Resolution

To resolve this issue, identify the specific component in an ERROR state and manually update its status to ACTIVE within the SDDC Manager platform database.

1. Identify the component(s) in SDDC Manager which is in ERROR state

   1. SSH to SDDC manager with vcf and su to root

   2. Run VCF VDT to identify the component in ERROR state. See Using the VCF Diagnostic Tool for SDDC Manager 

      INVENTORY STATUS
      
              [PASS]    Host Status Check
                          All hosts are in an ACTIVE state.
      
              [PASS]    Domain Status Check
                          All domains are in an ACTIVE state.
      
              [FAIL]    vCenter Status Check
                          ########-####-####-####-########4ce2 | vcsa01.example.com | ERROR
      
              [PASS]    PSCs Status Check
                          All PSCs are in an ACTIVE state.
      
              [PASS]    Cluster Status Check
                          All Clusters are in an ACTIVE state.
      
              [PASS]    NSX Manager Status Check
                          All NSX Managers are in an ACTIVE state.

2. Update the component status to ACTIVE in SDDC database
   
   

   1. Take Snapshot of the SDDC Manager VM

   2. SSH to SDDC manager as the vcf user and su to root.

   3. Connect to platform database
      For SDDC 4.x

      psql -h localhost -U postgres -d platform

      For SDDC 5.x:

      /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform

   4. Update the component status to ACTIVE identified in Step 1()

      update <database_table_name> set status='ACTIVE' where id='<id from step 1(b)>';

      Sample (vCenter):

      update vcenter set status='ACTIVE' where id='########-####-####-####-########4ce2';

      Sample (NSX):

      
      update nsxt set status='ACTIVE' where id='########-####-####-####-########e92d'; 

3. Retry the plan upgrade from the SDDC Manager UI.

4. (Optional) May need to restart the SDDC Manager services:

   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

    

Additional Information

Commands to check the specific tables manually:

• /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select * from domain where status !='ACTIVE'"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select * from host where status !='ACTIVE'"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select * from vcenter where status !='ACTIVE'"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select * from nsxt where status !='ACTIVE'"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select * from cluster where status !='ACTIVE'"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select id, lb_hostname, status from vrli;"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select id, lb_hostname, status from vrops;"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select id, vrslcm_node, status from vrslcm;"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select id,vm_hostname,status from psc;"
  /usr/bin/psql -h localhost -U postgres -d platform -c "\x" -c "select id,cluster_fqdn,status from nsxt;"

  
  
  

• /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "select * from sddc_manager_controller;"

  
  
  

• psql -h localhost -U postgres -d platform -c "select id, lb_hostname, status from vrli;"

• psql -h localhost -U postgres -d platform -c "select id, lb_hostname, status from vrops;"

• psql -h localhost -U postgres -d platform -c "select id, vrslcm_node, status from vrslcm;"