NSX upgrade from VCF Operations to 9.0.1 fails - Update not possible while [] is in failed state.

Products

• VMware vCenter Server

Issue/Introduction

• Upgrade of NSX manager from 9.0.0 to 9.0.1 shows an error on the VCF Ops

• If you navigate to Available updates the bundles in not available to proceed with the upgrades with the error - Update not possible while [] is in failed state. 

• The component in the error that reports the failed state can be the vCenter, NSXT manager or an ESXi host. For example in this case the vCenter server is reported in a failed state.

Environment

VCF 9.0

Cause

The vCenter server has been marked in an ERROR state from the SDDC manager Database. 

Resolution

1. Take snapshot of SDDC Manager VM.
   
   
2. Confirm the status of the vCenter server from the SDDC DB with the below command 
   
   psql -h localhost -U postgres -d platform -c "select id, status, vm_hostname from vcenter where status!='ACTIVE';"
   
   
3. Update the vCenter server status to ACTIVE.
   
   psql -h localhost -U postgres -d platform -c "update vcenter  set status='ACTIVE' where id='<id from step #2>';"


4. Restart the services on the SDDC
   
   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh


5. Retry the Upgrade from the VCF Operations UI. 

Additional Information

Using the VCF Diagnostic Tool for vSphere (VDT) we can identify the inventory state to check if a component is in ACTIVE or ERROR state. 

1. Download VCF Diagnostic Tool and copy it to SDDC Manager VM.
2. SSH to SDDC Manager VM with vcf user and elevate to root with su
3. Run VCF Diagnostic tool on SDDC Manager VM to identify the inventory in ERROR state - Refer Using the VCF Diagnostic Tool for SDDC Manager 
   
   Sample output:

      INVENTORY STATUS
   
           [FAIL]    vCenter Status Check
                       ####-####-#### | vcsa_fqdn | ERROR
   
           [FAIL]    Domain Status Check
                       ####-####-#### | domain_name | ERROR
   
           [PASS]    Host Status Check
                       All vCenters are in an ACTIVE state.
   
           [PASS]    PSCs Status Check
                       All PSCs are in an ACTIVE state.
   
           [PASS]    Cluster Status Check
                       All Clusters are in an ACTIVE state.
   
           [PASS]    NSX Manager Status Check
                       All NSX Managers are in an ACTIVE state.
   
           [PASS]    NSX Edge Status Check
                       All NSX Edges are in an ACTIVE state.

Based on what components are in ERROR state, the SDDC manager database should be updated to make the component into an ACTIVE state. Ensure to take a snapshot before the change.