SDDC Manager Precheck fails with error: "Check if out-of-band upgrade have been performed"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• You are attempting to upgrade VMware Cloud Foundation (VCF)
• During the pre-checks for an NSX update, the validation fails
• The SDDC Manager UI displays the following error : "Check if out-of-band upgrade have been performed"
• Reviewing the /var/log/vmware/vcf/operationsmanager/operationsmanager.log

• XXXX-XX-XXTXX:XX:XX.XXX+XXXX DEBUG [vcf_om,XXXXXX,XXXXXX] [c.v.v.b.p.t.v.v.ExternalValidationValidator,precheck-validation9] Running NATIVE validator out-of-band-upgrade-validator on domain entity with name XXXXXX - validation run id XXXXXX
  XXXX-XX-XXTXX:XX:XX.XXX+XXXX DEBUG [vcf_om,XXXXXX,XXXXXX] [c.v.v.b.c.v.OutOfBandUpgradeValidator,precheck-validation9] Out-of-band upgrade validation entities: [VCENTER, ESXI, NSXT_CLUSTER]
  XXXX-XX-XXTXX:XX:XX.XXX+XXXX DEBUG [vcf_om,XXXXXX,XXXXXX] [c.v.v.i.s.s.c.r.CredentialResource,precheck-validation9] There is no PSC associated with vcenter with resource name: XXXXXX. Defaulting to management VC's PSC SSO credential
  XXXX-XX-XXTXX:XX:XX.XXX+XXXX ERROR [vcf_om,XXXXXX,XXXXXX] [c.v.v.i.s.s.impl.EsxiVersionSync,precheck-validation9] Exception occurred when fetching SSO credentials for PSC
  com.vmware.vcf.inventory.sync.service.exception.InventorySyncException: vCenter SSO credentials doesnt exist for PSC {0}.

Environment

VMware Cloud Foundation (VCF) 5.2.x

VMware Cloud Foundation (VCF) 9.x

Cause

During prechecks, SDDC Manager attempts to look up the Single Sign-On (SSO) credentials for the shared Platform Services Controller (PSC).To determine the parent domain, the underlying code splits the Fully Qualified Domain Name (FQDN) by periods and removes only the first segment. While this works for standard naming conventions (where removing the first segment leaves identical parent domains, such as example.local), it breaks in environments where unique workload and management identifiers are placed in the second segment of the FQDN.

Example of the parsing failure:

Workload vCenter:vc01.wld01.example.local becomes wld01.example.local.  

Management vCenter: vc01.mgmt01.example.local becomes mgmt01.example.local.  

Because these resulting strings do not match, the fallback logic incorrectly assumes the domains are different and fails to retrieve the credential, even if it is healthy and present in the database. 

Resolution

Connect to the SDDC Manager appliance via SSH using the vcf user, and elevate your privileges to root.

Create a backup of the target configuration file before making any changes.

Open the following file in a text editor:

vi /opt/vmware/vcf/operationsmanager/scripts/assessment/perspectives/domain/domain-upgrade-perspective.json
 

Locate the out-of-band-upgrade-validator block within the externalValidations array.  

Add the line "runCondition": " false", to this block. It should look like this:  JSON"externalValidations":

[
  {
    "type": "NATIVE",
    "id": "out-of-band-upgrade-validator",
    "path": "out-of-band-upgrade-validator",
    "runCondition": " false",
    ...
  }
]

 Save and close the file.Restart the operationsmanager service to apply the configuration changes:

systemctl restart operationsmanager

 Monitor the Operations Manager local API to verify when the service comes back online.

Wait until the output indicates an ACTIVE state:

watch curl localhost/operationsmanager/about
 

Once the service is fully online, retry the prechecks from the SDDC Manager UI.