vCenter upgrade to version 8.0 fails with the error: "Upgrade phase timed out. The time planned for the upgrade phase was 15 minutes. The upgrade phase has already been running for 15 minutes."
search cancel

vCenter upgrade to version 8.0 fails with the error: "Upgrade phase timed out. The time planned for the upgrade phase was 15 minutes. The upgrade phase has already been running for 15 minutes."

book

Article ID: 401638

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vCenter upgrade pre-check failure at stage 2 is observed with the error: Upgrade phase timed out. The time planned for the upgrade phase was 15 minutes. The upgrade phase has already been running for 15 minutes.

 

  • /var/log/vmware/upgrade/requirements-upgrade-runner.log:

    YYYY-MM-DDTHH:MM:SS ERROR __main__ Distributed upgrade mode 'requirements' failed
    YYYY-MM-DDTHH:MM:SS INFO root Exiting with exit-code 1
    YYYY-MM-DDTHH:MM:SS WARNING status_reporting_sdk.msgL10n Got error and will use english message while formatting message: id=upgrade.vcIntegrity.warning.resolution, text='Please review VMware Lifecycle Manager % (0)s Documentation for details', args= {}
    YYYY-MM-DDTHH:MM:SS INFO wf.wf_processor State CollectRequirements com.vmware.vcdb has succeeded.
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states sso:CollectRequirements: didn't have progress. Next report will come after 60 seconds
    YYYY-MM-DDTHH:MM:SS WARNING wf.wf_processor Operation State CollectRequirements com.vmware.sso did not report progress for 901 seconds
    YYYY-MM-DDTHH:MM:SS INFO wf.wf_processor Stopping all running states
    YYYY-MM-DDTHH:MM:SS INFO wf.wf Stopping State CollectRequirements com.vmware.sso, represented by <wf.wf.StateWrapper object at 0x7fa0b4be1bd0>
    YYYY-MM-DDTHH:MM:SS INFO wf.wf Stopping State CollectRequirements com.vmware.sso, represented by <upgrade.workflow_builder._ProgressReportingState object at 0x7fa0b4be1a90>
    YYYY-MM-DDTHH:MM:SS INFO output.requirements_result_producer Setting storage partitions for component 'upgrade_framework'.
    YYYY-MM-DDTHH:MM:SS INFO output.requirements_result_producer Persisting preupgrade result :[
        {
            "name": "upgrade_framework",
            "installedOn": "localhost",
            "optional": false,
            "requirements": {
                "coreRequirement": {
                    "requiredSrcDiskSpace": 0.0,
                    "requiredDstDiskSpace": {},
                    "exportEstimationTime": 0,
                    "importEstimationTime": 0
                },
                "dstPortSpecs": [],
                "srcPorts": [],
                "userOptionSpecs": [],
                "installArguments": {},
                "extraArguments": {
                    "legacySsoFqdn": "####",
                    "legacySsoIps": [
                        "127.0.0.1"
                    ],
                    "legacySsoHttpsPort": 443,
                    "legacySsoAdminEndpoint": "/sso-adminserver/sdk/vsphere.local",
                    "legacyStsUrl": "https://####/sts/STSService/vsphere.local",
                    "legacyCpuCores": 4
                },
                "requirementMismatchSpecs": [
                    {
                        "text": {
                            "id": "ur.upgrade.execution.timedout.text",
                            "translatable": "Upgrade phase timed out. The time planned for the upgrade phase was %(0)s minutes. The upgrade phase has already been running for %(1)s minutes.",
                            "args": [
                                15,
                                15
                            ],
                            "localized": "Upgrade phase timed out. The time planned for the upgrade phase was 15 minutes. The upgrade phase has already been running for 15 minutes."
                        },
                        "description": null,
                        "resolution": {
                            "id": "ur.upgrade.execution.timedout.resolution",
                            "translatable": "To extend the default timeout, set environment variable % (0)s for export upgrade phase, % (1)s for import upgrade phase, and %(2)s for import historical data phase. The timeout is in minutes.",
                            "args": [

                                "UPGRADE_EXPORT_TIMEOUT",
                                "UPGRADE_IMPORT_TIMEOUT",
                                "UPGRADE_POST_IMPORT_TIMEOUT"
                            ],
                            "localized": "To extend the default timeout, set environment variable UPGRADE_EXPORT_TIMEOUT for export upgrade phase, UPGRADE_IMPORT_TIMEOUT for import upgrade phase, and  UPGRADE_POST_IMPORT_TIMEOUT for import historical data phase. The timeout is in minutes."
                        },
                        "problemId": null,
                        "severity": "ERROR"
                    }
                ]
            }
        }
    ]

  • Identify the tenant's name from the output of the command via VCSA ssh session:

    root@vcenter [ / ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost
    vsphere.local
  • sso-config command to fetch the authentication policy on vCenter is hung:

    root@vcenter [ / ]# sso-config.sh -get_authn_policy -t vsphere.local
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-slf4j-impl-2.21.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/vmware/lib64/log4j-slf4j-impl-2.21.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    Authentication policy on tenant : 'vsphere.local' :
    IsPasswordAuthEnabled:  true
    IsWindowsAuthEnabled:  true
    IsTLSClientCertAuthnEnabled:  false
    IsSecurIDAuthnEnabled:  false
    revocationCheckEnabled:  true
    useOCSP:  false
    useCRLAsFailOver:  false
    Alternative OCSP responders:
    useCertCRL:  true
    CRLUrl:   UndefinedConfig
    Exception in thread "main" java.lang.NoSuchMethodError: com.vmware.identity.idm.ClientCertPolicy.getEnableHint()Z
            at com.vmware.identity.ssoconfig.SsoConfig.displayAuthenticationPolicy(SsoConfig.java:1064)
            at com.vmware.identity.ssoconfig.SsoConfig.get_authn_policy(SsoConfig.java:1000)
            at com.vmware.identity.ssoconfig.SsoConfig.main(SsoConfig.java:542)

Environment

VMware vCenter Server 7.0.x

Cause

The CollectRequirements.py calls the sso-config.sh command. So, we can see the command is giving NoSuchMethodError in the affected environment, which is the cause of 15 minutes timeout.

Resolution

To workaround the issue, perform the below steps

  • Log in to Broadcom Support portal web page.
  • Download the Full-Patch ISO for the same source build from the download page. Refer to VMware vSphere downloads, VMware Converter, OEM custom images, patches and addons in the Broadcom Support Portal
  • Extract or Mount the file on a client machine
  • Copy the rpm "vmware-identity-sts-<version>-<build>.noarch.rpm" file to the affected vCenter server
  • Take a proper backup of the VCSA before proceeding
  • Manually install the rpm on the VCSA via the below command

    Sample Command:
    rpm -Uvh --force vmware-identity-sts-7.0.3.27000-14372954.noarch.rpm
  • Restart all the services on the vCenter: service-control --stop --all && service-control --start --all
  • Proceed with the upgrading the vCenter server

Additional Information

Powered by Wolken Software