vCenter Upgrade fails with "internal error occurs during execution of upgrade" during pre-checks

Products

• VMware vCenter Server
• VMware vCenter Server 6.0
• VMware vCenter Server 7.0

Issue/Introduction

Symptoms:

• vCenter upgrade fails with internal errors during pre-checks in upgrade stage 2  caused by stale extension com.vmware.migrate-connector.127.0.0.1 from a previous migration/upgrade.
• In /var/log/vmware/upgrade/requirements-upgrade-runner.log you see the following error:
  
  

YYYY-MM-DDThh:mm:ssZ INFO output.requirements_result_producer Persisting preupgrade result :[
    {
        "requirements": {
            "installArguments": {},
            "userOptionSpecs": [],
            "coreRequirement": {
                "requiredSrcDiskSpace": 0.0,
                "requiredDstDiskSpace": {},
                "exportEstimationTime": 0,
                "importEstimationTime": 0
            },
            "requirementMismatchSpecs": [
                {
                    "description": null,
                    "severity": "ERROR",
                    "text": {
                        "localized": "Internal error occurs during execution of upgrade process.",
                        "translatable": "Internal error occurs during execution of upgrade process.",
                        "id": "ur.internal.text"
                    },
                    "resolution": {
                        "localized": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",
                        "translatable": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",
                        "id": "ur.internal.resolution"
                    },
                    "problemId": null
                }
            ],
            "dstPortSpecs": [],
            "extraArguments": {
                "legacySsoIps": null,
                "legacySsoFqdn": null,
                "legacyStsUrl": null,
                "legacySsoAdminEndpoint": null,
                "legacySsoHttpsPort": null,
                "legacyCpuCores": 4
            },
            "srcPorts": []
        },
        "name": "upgrade_framework",
        "optional": false,
        "installedOn": "localhost"
    }
]



YYYY-MM-DDThh:mm:ssZ INFO deployer.migration_env_deployer Updating extension -- com.vmware.migrate-connector.127.0.0.1
YYYY-MM-DDThh:mm:ssZ ERROR UpgradeRunner Upgrade Runner has encountered an exception
Traceback (most recent call last):
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/UpgradeRunner.py", line 1889, in main
    envDeployer.setup(srcTargets, destTargets, isRegVcExt)
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/py/deployer/migration_env_deployer.py", line 563, in setup
    self._setupTargets(srcTargetsToSetup, False, isRegVcExt)
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/py/deployer/migration_env_deployer.py", line 535, in _setupTargets
    self._registerTargetAsExtension(targetConfig)
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/py/deployer/migration_env_deployer.py", line 345, in _registerTargetAsExtension
    _registerExtension(srcSi, extKey, extCertificate)
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/py/deployer/migration_env_deployer.py", line 78, in _registerExtension
    em.UpdateExtension(ext)
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 556, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 368, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/tmp/vmware-upgrade-temp-dir59Ig5fFWHm/tmp7d8uBv3e9X/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 1481, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'lookup.fault.EntryNotFoundFault',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   reason = 'Invalid fault'
}

Environment

VMware vCenter Server Appliance 6.7.x

VMware vCenter Server 7.0.x

Cause

The "com.vmware.migrate-connector.127.0.0.1" is an extension that is registered to the appliance during migration/upgrade process. It is expected to automatically remove itself once the process is complete, however in this case it seems like in a previous migration/upgrade, the extension was not removed. This could cause issue in a subsequent upgrade during pre-checks.

Resolution

Unregister the extension "com.vmware.migrate-connector.127.0.0.1" from vCenter MOB (Managed Object Browser) page.

Note: If vCenter Server is in linked mode, perform this procedure on the vCenter Server that is used to install the plugin initially, then restart the vCenter Server services on the linked vCenter Server.

To resolve this issue, remove unwanted plugin from the available plugin list using vCenter Manage Object page.

In a web browser, navigate to https://vCenter_Server_name_or_IP/mob.

Where vCenter_Server_name_or_IP/mob is the name of the vCenter Server or its IP address. Log in with the vCenter SSO admin credentials.

Note: If you are receiving "HTTP Status 401 - Unauthorized" error in the browser, please refer this KB - "HTTP Status 401 - Unauthorized", accessing vCenter MOB fails from Edge & Chrome browsers
 
->Click Content.
->Click ExtensionManager.
->Select and copy the name of the plug-in to remove from the list of values under Properties. For a list of default plugins, see the Additional Information section of this article.
->Click UnregisterExtension. A new window appears.
->Paste the key of the plugin and click Invoke Method. This removes the plugin and results in void.
->Close the window.
->Refresh the Managed Object Type:ManagedObjectReference:ExtensionManager window to verify that the plugin is removed successfully.
Note: If the plugin still appears in UI, a restart of the vSphere Client may be required.

 
To disable the plugin through the Client PlugIn manager within the vSphere Web Client 5.1 and later:

->Log in to the vSphere Web Client with an vCenter Server Administrative user.
->When the error appears, click No.
->Navigate to Administration > Client Plug-Ins.
->Locate the client plug-in to disable.
->Right-click this entry and select Disable.
->Click Yes on the pop-up box for disabling the plugin.

Refer KB : Cannot remove or disable unwanted plug-ins from vCenter Server and vCenter Server Appliance

Additional Information

Impact/Risks:

No impact, however taking snapshot of the vCenter before making any changes is recommended.