vCenter upgrade from 8.x to 9.x via VCF Operations fails with error: START_ASSESSMENT_FAILED Failed to find SDDC Manager SSO credential or BASELINER_DATASOURCE_SDDC_MANAGER_CREDENTIALS_MISSING

Products

• VCF Operations
• VMware SDDC Manager / VCF Installer
• VMware vCenter Server

Issue/Introduction

• Under VCF Operations UI, vCenter upgrade fails at VCENTER UPGRADE PRECHECK stage.
  
  
  
  
• Under the Task Info, below error message is available:
  
  
  
  

• Under SDDC's /var/log/vmware/vcf/lcm/lcm-debug.log, below log snips are found:

  ########","autoDownloadEnabled":true,"pollingInterval":########, "outOfDateWarningThreshold":60,"outOfDateErrorThreshold":90,"updateTime": ########, "pollingEnabled": true}
YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm,########,####] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-####-exect-#] [TOKEN] START_ASSESSMENT_FAILED Failed to find SDDC Manager SSO credential.
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to find SDDC Manager SSO credential.
at com.vmware.evo.sddc.lcm.services. impl.AssessmentPrechecksServiceImpl.toLcmException(AssessmentPrechecksServiceImpl.java:264)
at com.vmware.evo.sddc.lcm.services. impl.AssessmentPrechecksServiceImpl.runAssessment(AssessmentPrechecksServiceImpl.java:137)
  
  

• Under SDDC's /var/log/vmware/vcf/lcm/vcenter/######-####-####-####-######/precheck/workflow_########/vcsa-cli-installer.log, below log snips are found:

  YYYY-MM-DD HH:MM:SS - vCSACliInstallLogger - ERROR - Template structure validation failed for template /var/log/vmware/vcf/lcm/vcenter/######-####-####-####-######.input.json.
YYYY-MM-DD HH:MM:SS - vCSACliInstallLogger - ERROR - The value '<user-ID-vCenter-fqdn@sso-domain>' of the key 'username' in section 'source_vc', subsection 'vc_vcsa' is invalid. Correct the value and rerun the script.
YYYY-MM-DD HH:MM:SS - vCSACliInstallLogger - INFO -[FAILED] Task: StructureValidationTask: Executing Template Structure Validation task execution failed at HH:MM:SS =======
YYYY-MM-DD HH:MM:SS - vCSACliInstallLogger - DEBUG - Task 'StructureValidationTask: Executing Template Structure Validation task' execution failed because [Template structure validation failed for template /var
/log/vmware/vcf/lcm/vcenter/######-####-####-####-######/input.json.], possible resolution is [You must correct the value errors in the template '/var/log/vmware/vcf/lcm/vcenter/######-####-####-####-######/input.json'.]

  YYYY-MM-DD HH:MM:SS - vCSACliInstallLogger - ERROR - Error message: Template structure validation failed for template /var/log/vmware/vcf/lcm/vcenter/######-####-####-####-######/input.json.
YYYY-MM-DD HH:MM:SS - vCSACliInstallLogger - INFO - Result and Log File Information ...
YYYY-MM-DD HH:MM:SS - vCSACliInstallLogger - INFO - WorkFlow log directory: /var/log/vmware/vcf/lcm/vcenter/######-####-####-####-######/precheck/workflow_########

• Under SDDC's /var/log/vmware/vcf/operationsmanager/operationsmanager.log, below log snips are found:
  
  YYYY-MM-DDTHH:MM:SS INFO [vcf_om,##########,####] [o.b. jsse.provider. PropertyUtils, ForkJoinPool. commonPool-worker-14] Found string system property [java. home]: /usr/lib/jvm/openjdk-java17-headless.x86_64
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,########,####] [c.v.v.b.i.p.h.AbstractInternalDatasourceHandler,http-nio-127.0.0.1-####-exect-#] Retrieved service account credentials for entityId ########-####-####-####-########, entityType ESXI, credentialType SSH - count is 1
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,########,####] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-####-exect-#] Processing localizable exception Failed to find SDDC Manager SSO credential.
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,########,####] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-####-exect-#] [[TOKEN]] BASELINER_DATASOURCE_SDDC_MANAGER_CREDENTIALS_MISSING Failed to find SDDC Manager SSO credential.
com.vmware.evo.sddc.orchestrator. exceptions.OrchTaskException: Failed to find SDDC Manager SSO credential.
at com.vmware. vcf.baseliner. input.provider.handler. AbstractInternalDatasourceHandler. lambda$getSddcManagerCredData$4(AbstractInternalDatasourceHandler. java: 152)
  
  

Environment

• VCF Operations 9.x
• SDDC Manager 9.x
• vCenter 8.x/9.x

Cause

The upgrade precheck fails because the input.json file generated for the upgrade contains an incorrect or non-existent User ID for the source vCenter. Specifically, the automation fails to map the correct service account credentials (svc-vcf-installer), leading to a validation error when the VCSA CLI installer attempts to parse the upgrade template.

Resolution

To resolve this issue, follow the below steps:

1. Take a virtual machine snapshot of the SDDC Manager appliance.
2. Log in to the SDDC Manager via SSH using the vcf user and switch to root.
3. Check the latest vcsa-cli-installer.log to identify the specific path of the failed input.json:
   
   Example: user-ID-vCenter-fqdn@sso-domain from the above logs.
   
   
4. Edit the input.json file from the path: /var/log/vmware/vcf/lcm/vcenter/<vCenter-UUID>/input.json, using below command:
   
   vi /var/log/vmware/vcf/lcm/vcenter/<vCenter-UUID>/input.json
   
   
5. Locate the user that was found under /var/log/vmware/vcf/lcm/vcenter/########-####-####-####-########/########-####-####-####-########/precheck/workflow_########/vcsa-cli-installer.log and replace it with svc-vcf-installer-<vCenter-fqdn@sso-domain, wherever applicable.
   
   Example: Change user-ID-vCenter-fqdn@sso-domain to svc-vcf-installer-<vCenter-fqdn@sso-domain
   
   
6. Press Esc + wq! and hit Enter to save and exit the file edit.
7. Return to the VCF Operations Manager UI and proceed with the upgrade.