vCenter Server 6.7 upgrade to 7.0.x fails with error "internal error occurs during execution of upgrade" during pre-checks

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter Server upgrade fails from 6.7 to 7.0 at stage 2 (during upgrade prechecks) with the below error:

 

internal error occurs during execution of upgrade

• You may see similar entries in the below log files:

/var/log/vmware/applmgmt/applmgmt.log
2022-02-17T23:08:28 PM UTC [6655]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login          Failures Latest failure    From\nroot               0   \n' 2022-02-17T23:08:34 PM UTC [6655]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.vcenter.deployment.upgrade, operation_id: check 2022-02-17T23:08:34 PM UTC [6655]DEBUG:vmware.vherd.base.authorization_local:Local authorization initialized 2022-02-17T23:08:34 PM UTC [6655]ERROR:vmware.appliance.extensions.authorization.authorization_sso:Failed to get certificate or key from VECS Traceback (most recent call last):  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 424, in __init__    applmgmt_ks.load('machine')  File "/usr/lib/vmware/site-packages/identity/vmkeystore.py", line 99, in load    self._store_context = self._client.OpenCertStore(store_name, password) RuntimeError: unidentifiable C++ exception 2022-02-17T23:08:34 PM UTC [6655]ERROR:root:Authorization module (authorization_sso) failed to initialize {unidentifiable C++ exception}

/var/log/vmware/applmgmt/vami.log
2022-02-17T23:08:26.270 [8030]ERROR:detwist:Method failure = <twisted.python.failure.Failure vmware.vherd.base.pint.Error: pint.Error('com.vmware.applmgmt.err_status_does_not_exist', "Status for '%(0)s' does not exist.", **{'args': 'RESTORE'})> 2022-02-17T23:08:26.314 [8030]ERROR:detwist:Traceback (most recent call last):  File
[..]
"/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/impl.py", line 1162, in get   args=op) vmware.vherd.base.pint.Error: pint.Error('com.vmware.applmgmt.err_status_does_not_exist', "Status for '%(0)s' does not exist.", **{'args': 'RESTORE'})

/var/log/vmware/upgrade/upgrade-requirements.log
2022-02-17T23:08:50.985Z ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.log Traceback (most recent call last):  File "bootstrap_scripts/run-preupgrade-checks.py", line 145, in main    commandFactory = getCommandFactory(upgradeParameters, execData, reporter)  File 
[..]
"/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 1574, in InvokeMethod    "%d %s" % (resp.status, resp.reason)) http.client.HTTPException: 503 Service Unavailable 2022-02-17T23:08:50.986Z INFO root Exiting with exit-code 1

• The file /var/log/vmware/upgrade/requirements-upgrade-runner.log doesn't exist.

Environment

VMware vCenter Server Appliance 6.7.x

Resolution

To fix this issue, please follow the below steps:

1. Take an offline snapshot (vCenter VM powered off) of vCenter Server ( All nodes in case of Enhanced Linked Mode).
2. Power on vCenter VM and make sure vCenter UI is accessible and inventory is loaded successfully.
3. Run the fixsts.sh script. For more information, see "Signing certificate is not valid" error in VCSA 6.5.x,6.7.x or vCenter Server 7.0.x (76719)
4. Try the vCenter upgrade again.