Pre-Check error: "Internal error occurs during execution of upgrade process" when upgrading vCenter server from 7.0 to 8.0
search cancel

Pre-Check error: "Internal error occurs during execution of upgrade process" when upgrading vCenter server from 7.0 to 8.0

book

Article ID: 344940

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Upgrading vCenter Server Appliance (VCSA) fails during precheck with the error "Internal error occurs during execution of upgrade process"

  • Below error is seen in /var/log/vmware/upgrade/requirements-upgrade-runner.log: (target appliance)

[YYYY-MM-DDTHH:MM:SS] INFO deployer.migration_env_deployer Updating extension -- com.vmware.migrate-connector.127.0.0.1
[YYYY-MM-DDTHH:MM:SS] ERROR UpgradeRunner Upgrade Runner has encountered an exception
Traceback (most recent call last):
  self._setupTargets(srcTargetsToSetup, False, isRegVcExt)
  File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/py/deployer/migration_env_deployer.py", line 535, in _setupTargets
    self._registerTargetAsExtension(targetConfig)
  File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/py/deployer/migration_env_deployer.py", line 345, in _registerTargetAsExtension
    _registerExtension(srcSi, extKey, extCertificate)
  File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/py/deployer/migration_env_deployer.py", line 78, in _registerExtension
    em.UpdateExtension(ext)
  File "/tmp/vmware-upgrade-temp-dircnqp0mdxVp/tmpvEWX5jFuux/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", 
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'lookup.fault.EntryNotFoundFault',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   reason = 'Invalid fault'

[YYYY-MM-DDTHH:MM:SS] INFO output.requirements_result_producer Setting storage partitions for component 'upgrade_framework'.
[YYYY-MM-DDTHH:MM:SS] INFO output.requirements_result_producer Persisting preupgrade result :[

            },
            "requirementMismatchSpecs": [
                {
                    "text": {
                        "id": "ur.internal.text",
                        "translatable": "Internal error occurs during execution of upgrade process.",
                        "localized": "Internal error occurs during execution of upgrade process."
/var/log/vmware/lookupsvc/lookupserver-default.log: (source vCenter server)
[YYYY-MM-DDTHH:MM:SS] pool-2-thread-3                                                           INFO  com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Domain: vsphere.local}
[YYYY-MM-DDTHH:MM:SS] pool-2-thread-5                                                           INFO  com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation set is permitted for user {Name: machine-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Domain: vsphere.local}
[YYYY-MM-DDTHH:MM:SS] pool-2-thread-5                                                           INFO  com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation set is permitted for user {Name: vpxd-extension-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Domain: vsphere.local}

 

  • When upgrading vCenter via CLI installer, the following output is seen.

    Pre-upgrade check error: 1. Title: Internal error occurs during execution of upgrade process.
Suggested Resolution: Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.
Basic requirements:
Estimated disk space required on source VM for export data: 0.0 GB
Estimated total disk space required on target VM: 0 GB
No problem ID available for the error. Setting it to ''
================ [FAILED] Task: SrcRequirementTask: Running SrcRequirementTask execution failed at [HH:MM:SS] ================
================================================================================
Error message: Errors encountered during the requirements verification process.
Title: Internal error occurs during execution of upgrade process. 
Suggested Resolution: Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

Environment

VMware vCenter Server 8.0

VMware vCenter Server 7.0

Cause

  • This issue is seen if the SSO domain name does not correspond with the solutionUser section in vpxd.cfg. This situation can occur if the domain name was altered using: cmsso-util domain-repoint.

  • The SSO domain name is incorrectly set to "vshpere.local" instead of "vsphere.local."

Example: /etc/vmware-vpx/vpxd.cfg

<solutionUser>
  <certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
  <name>vpxd-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@vshpere.local</name>
  <privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
</solutionUser>

  • Additionally, the extension com.vmware.migrate-connector might exist as a stale extension.

Resolution

This issue is resolved in vCenter Server 8.0 Update 3

To work around this issue:

  • Update the vpxd.cfg with the correct SSO domain name

    • Log in to the source VCSA

    • Make a note of domain name from the command: /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

    • Execute the command to update the vpxd.cfg (Backup the vpxd.cfg file before performing the action plan.): sed -i 's/<current value>/<output from previous step>/g' /etc/vmware-vpx/vpxd.cfg

Example:
sed -i 's/vshpere.local/vsphere.local/g' /etc/vmware-vpx/vpxd.cfg

  • Restart the vpxd service : service-control --restart vmware-vpxd

  • Delete the "com.vmware.migrate-connector.127.0.0.1" via the vCenter server MOB using the steps mentioned in Cannot remove or disable unwanted plug-ins from vCenter Server and vCenter Server Appliance

  • If the migration connector is not listed in the MOB, validate if it is present in VCDB using the below command:

    • /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c  "select ext_id, version, desc_summary, company from vpx_ext;"

                        ext_id                 |  company   |    desc_summary   
----------------------------------------+---------+-------------------------+--------------+------------
 com.vmware.migrate-connector.127.0.0.1 | VMware, Inc. | Migration connector |

  • If migrator-connector is present in VCDB, Delete it from VCDB:  

    • Stop the vpxd service: service-control --stop vmware-vpxd

    • Delete migrator-connector : 

      • /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "delete from vpx_ext where ext_id = 'com.vmware.migrate-connector.127.0.0.1';"

    • Start the vpxd service: service-control --start vmware-vpxd

Additional Information

  • Refer the KB to unregister the extension "com.vmware.migrate-connector.127.0.0.1" : vCenter Upgrade fails with "internal error occurs during execution of upgrade" during pre-checks
  • "com.vmware.migrate-connector.127.0.0.1" is an extension that is registered to the appliance during migration/upgrade process. It is expected to automatically remove itself once the process is complete, however in this case it seems like in a previous migration/upgrade, the extension was not removed. This could cause issue in a subsequent upgrade during pre-checks.
Powered by Wolken Software