Upgrading vcenter to 7.0 fails at stage 2 with precheck error "Internal error occurs during execution of upgrade process"
search cancel

Upgrading vcenter to 7.0 fails at stage 2 with precheck error "Internal error occurs during execution of upgrade process"

book

Article ID: 316345

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • Pre-check error at stage 2: "Internal error occurs during execution of upgrade process."
  • Below errors are seen in logs:
    • /var/log/vmware/upgrade/requirements-upgrade-runner.log (destination VC)  
      YYYY-MM-DDTHH:MM:SSZ INFO deployer.migration_env_deployer Installing extension -- com.vmware.migrate-connector.127.0.0.1
YYYY-MM-DDTHH:MM:SSZ ERROR deployer.migration_env_deployer General deployment error
Traceback (most recent call last):
  File "/tmp/vmware-upgrade-temp-dir4mUgMXnAt0/tmpwG3wNKcvhL/py/deployer/migration_env_deployer.py", line 82, in _registerExtension
    em.RegisterExtension(ext)
  File "/tmp/vmware-upgrade-temp-dir4mUgMXnAt0/tmpwG3wNKcvhL/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 592, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/tmp/vmware-upgrade-temp-dir4mUgMXnAt0/tmpwG3wNKcvhL/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 381, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/tmp/vmware-upgrade-temp-dir4mUgMXnAt0/tmpwG3wNKcvhL/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 1608, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Access to perform the operation was denied.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
}
YYYY-MM-DDTHH:MM:SSZ ERROR UpgradeRunner Cannot continue upgrade story because of -- General error: (vmodl.fault.SecurityError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Access to perform the operation was denied.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
} (cause VC_GENERAL_ERROR)
YYYY-MM-DDTHH:MM:SSZ INFO output.requirements_result_producer Setting storage partitions for component 'upgrade_framework'.
YYYY-MM-DDTHH:MM:SSZ INFO output.requirements_result_producer Persisting preupgrade result :[
    {
        "name": "upgrade_framework",
        "requirements": {
            "userOptionSpecs": [],
            "installArguments": {},
            "srcPorts": [],
            "extraArguments": {
                "legacySsoAdminEndpoint": null,
                "legacyStsUrl": null,
                "legacySsoFqdn": null,
                "legacySsoHttpsPort": null,
                "legacyCpuCores": 12,
                "legacySsoIps": null
            },
            "coreRequirement": {
                "requiredDstDiskSpace": {},
                "importEstimationTime": 0,
                "exportEstimationTime": 0,
                "requiredSrcDiskSpace": 0.0
            },
            "requirementMismatchSpecs": [
                {
                    "problemId": null,
                    "text": {
                        "localized": "Internal error occurs during execution of upgrade process.",
                        "translatable": "Internal error occurs during execution of upgrade process.",
                        "id": "ur.internal.text"
                    },
                    "resolution": {
                        "localized": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",
                        "translatable": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",
                        "id": "ur.internal.resolution"
                    },
                    "severity": "ERROR",
                    "description": null
                }
            ],
            "dstPortSpecs": []
        },
        "installedOn": "localhost",
        "optional": false
    }
]
    • /var/log/vmware/vpxdvpxd.log (source VC) 
      YYYY-MM-DDTHH:MM:SSZ info vpxd[22193] [Originator@6876 sub=Default opID=4ae70e1d] [VpxLRO] -- ERROR lro-584131739 -- ExtensionManager -- vim.ExtensionManager.registerExtension: vmodl.fault.SecurityError:
--> Result:
--> (vmodl.fault.SecurityError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = "Received SOAP response fault from [<cs p:00007fea30003670, TCP:Example.com:443>]: create
--> "
--> }
--> Args:
-->
--> Arg extension:
--> (vim.Extension) {
--> description = (vim.Description) {
--> label = "MigrationExtension",
--> summary = "Migration connector"
--> },
--> key = "com.vmware.migrate-connector.127.0.0.1",
--> company = "VMware, Inc.",
--> type = <unset>,
--> version = "7.0",
--> subjectName = <unset>,
--> server = <unset>,
--> client = <unset>,
--> taskList = <unset>,
--> eventList = <unset>,
--> faultList = <unset>,
--> privilegeList = <unset>,
--> resourceList = <unset>,
--> lastHeartbeatTime = "YYYY-MM-DDTHH:MM:SSZ",
--> healthInfo = (vim.Extension.HealthInfo) null,
--> ovfConsumerInfo = (vim.Extension.OvfConsumerInfo) null,
--> extendedProductInfo = (vim.ext.ExtendedProductInfo) null,
--> managedEntityInfo = <unset>,
--> shownInSolutionManager = <unset>,
--> solutionManagerInfo = (vim.ext.SolutionManagerInfo) null
--> }

       

    • /var/log/vmware/sts/vmware-sts-idmd.log : 
      YYYY-MM-DDTHH:MM:HHZ pool-2-thread-9010 vsphere.local        yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy INFO  com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] Cannot find solution user [vpxd-########-####-####-####-############@yourdomain.local] in [CN=ServicePrincipals,DC=vsphere,DC=local]

Environment

VMware vCenter Server 6.7
VMware vCenter Server 7.x

Cause

The Solution user mentioned in vmware-sts-idmd.log logs is incorrect.

Resolution

To resolve the issue update the correct solution user in vpxd.cfg:

NOTE: Before applying the steps below, please take a backup or an offline-snapshot (in powered-off state) of the vCenter Server Appliance . If the vCenter is part of a Linked Mode replication setup, please backup/snapshot all replicating nodes as well. Please refer: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

  • Copy the correct solution user from the command: /usr/lib/vmware-vmafd/bin/vmafd-cli  get-machine-id --server-name localhost
  • Open /etc/vmware-vpx/vpxd.cfg and update the correct solution user in the below line: 
    <solutionUser>
<certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
<name>vpxd-########-####-####-####-############@yourdomain.local</name>
<privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
</solutionUser>
  • Restart vCenter services: service-control --stop --all && service-control --start --all 

Powered by Wolken Software