Pre-Check error: "Internal error occurs during execution of upgrade process" when upgrading vCenter server from 7.0 to 8.0

Products

VMware vCenter Server

Issue/Introduction

Upgrading vCenter Server Appliance (VCSA) fails during precheck with the error "Internal error occurs during execution of upgrade process"
/var/log/vmware/upgrade/requirements-upgrade-runner.log: (target appliance)

[YYYY-MM-DDTHH:MM:SS] INFO deployer.migration_env_deployer Updating extension -- com.vmware.migrate-connector.127.0.0.1
[YYYY-MM-DDTHH:MM:SS] ERROR UpgradeRunner Upgrade Runner has encountered an exception
Traceback (most recent call last):
self._setupTargets(srcTargetsToSetup, False, isRegVcExt)
File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/py/deployer/migration_env_deployer.py", line 535, in _setupTargets
self._registerTargetAsExtension(targetConfig)
File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/py/deployer/migration_env_deployer.py", line 345, in _registerTargetAsExtension
_registerExtension(srcSi, extKey, extCertificate)
File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/py/deployer/migration_env_deployer.py", line 78, in _registerExtension
em.UpdateExtension(ext)
File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py",
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'lookup.fault.EntryNotFoundFault',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = 'Invalid fault'

[YYYY-MM-DDTHH:MM:SS] INFO output.requirements_result_producer Setting storage partitions for component 'upgrade_framework'.
[YYYY-MM-DDTHH:MM:SS] INFO output.requirements_result_producer Persisting preupgrade result :[

},
"requirementMismatchSpecs": [
{
"text": {
"id": "ur.internal.text",
"translatable": "Internal error occurs during execution of upgrade process.",
"localized": "Internal error occurs during execution of upgrade process."
/var/log/vmware/lookupsvc/lookupserver-default.log: (source vCenter server)
[YYYY-MM-DDTHH:MM:SS] pool-2-thread-3 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Domain: vsphere.local}
[YYYY-MM-DDTHH:MM:SS] pool-2-thread-5 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation set is permitted for user {Name: machine-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Domain: vsphere.local}
[YYYY-MM-DDTHH:MM:SS] pool-2-thread-5 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation set is permitted for user {Name: vpxd-extension-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Domain: vsphere.local}

When upgrading vCenter via CLI installer, the following output is seen.

Pre-upgrade check error: 1. Title: Internal error occurs during execution of
upgrade process.
Pre-upgrade check error: 1. Suggested Resolution: Please search for these
symptoms in the VMware Knowledge Base for any known issues and possible
resolutions. If none can be found, collect a support bundle and open a support
request.
Basic requirements:
Estimated disk space required on source VM for export data: 0.0 GB
Estimated total disk space required on target VM: 0 GB
No problem ID available for the error. Setting it to ''
================ [FAILED] Task: SrcRequirementTask: Running SrcRequirementTask
execution failed at [HH:MM:SS] ================
================================================================================
Error message: Errors encountered during the requirements verification process:
Title: Internal error occurs during execution of upgrade process. Suggested
Resolution: Please search for these symptoms in the VMware Knowledge Base for
any known issues and possible resolutions. If none can be found, collect a
support bundle and open a support request.

Environment

VMware vCenter Server 8.0

VMware vCenter Server 7.0

Cause

This issue can be caused if the SSO domain name does not match under solutionUser section in vpxd.cfg.

SSO domain name is mismatched "vshpere.local" instead of "vsphere.local."

Example:

/etc/vmware-vpx/vpxd.cfg

<solutionUser>
<certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
<name>vpxd-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@vshpere.local</name>
<privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
</solutionUser>

Additionally, the extension com.vmware.migrate-connector is a stale extension.

Resolution

This issue is resolved in vCenter Server 8.0 Update 3

To workaround this issue:

Update the vpxd.cfg with the correct SSO domain name

- Log in to the source VCSA
- Capture the output of the below command

/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

- Execute the below command to update the vpxd.cfg

sed -i 's/<current value>/<output from previous step>/g' /etc/vmware-vpx/vpxd.cfg

Example:
sed -i 's/vshpere.local/vsphere.local/g' /etc/vmware-vpx/vpxd.cfg

Delete the "com.vmware.migrate-connector.127.0.0.1" via the vCenter server MOB using the steps mentioned in Cannot remove or disable unwanted plug-ins from vCenter Server and vCenter Server Appliance
Restart the vpxd service

service-control --restart vpxd

Additional Information

See https://knowledge.broadcom.com/external/article/345216/vcenter-upgrade-fails-with-internal-erro.html

"com.vmware.migrate-connector.127.0.0.1" is an extension that is registered to the appliance during migration/upgrade process. It is expected to automatically remove itself once the process is complete, however in this case it seems like in a previous migration/upgrade, the extension was not removed. This could cause issue in a subsequent upgrade during pre-checks.

Unregister the extension "com.vmware.migrate-connector.127.0.0.1" from vCenter MOB (Managed Object Browser) page.