vCenter patching fails with error 'Internal error occurs during execution of update process'

Products

• VMware vCenter Server

Issue/Introduction

vCenter Server patching fails in pre-check phase.

Attempt to patch vCenter via VAMI or CLI, will fail with similar error.

The /var/log/vmware/applmgmt/PatchRunner log shows error similar to this below. 

####:##:## ##:##:##,081.81Z perfcharts:ComponentDiscovery ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'perfcharts:ComponentDiscovery' failed.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatey1ugsuvb/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/updatemgr/software-updatey1ugsuvb/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/updatemgr/software-updatey1ugsuvb/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/updatemgr/software-updatey1ugsuvb/stage/scripts/patches/payload/components-script/perfcharts/__init__.py", line 50, in discover
    if not is_patch_needed(perfcharts_props_file):
  File "/storage/updatemgr/software-updatey1ugsuvb/stage/scripts/patches/payload/components-script/perfcharts/__init__.py", line 121, in is_patch_needed
    mod = __import__(modulePath)
  File "/storage/updatemgr/software-updatey1ugsuvb/stage/scripts/patches/payload/components-script/perfcharts/patches/patch_01.py", line 24, in <module>
    from cloudvmcisreg import *
  File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 31, in <module>
    from cis.cisreglib import do_lsauthz_operation, parse_cisreg_prop, VmafdClient
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 30, in <module>
    from pyVmomi import (lookup, SoapStubAdapter, vmodl, dataservice,
ImportError: cannot import name 'lookup' from 'pyVmomi' (/usr/lib/python3.7/site-packages/pyvmomi-8.0.2.0.1-py3.7.egg/pyVmomi/__init__.py)
.
####:##:## ##:##:##.91Z ERROR __main__ Discovery of vCSA patching components failed

Similar messages may also be seen in the /var/log/vmware/applmgmt/software-packages.log


####:##:## ##:##:##.197 [32330]DEBUG:vmware.appliance.update.update_b2b:Non-leaf rpm found: 'libgpg-error'
####:##:## ##:##:##.197 [32330]ERROR:vmware.appliance.update.update_functions:Can't read JSON file /storage/core/software-update/install_operation [Errno 2] No such file or directory: '/storage/core/software-update/install_operation'
####:##:## ##:##:##.197 [32330]DEBUG:vmware.appliance.update.update_functions:Exception happens while waiting for task out file to appear: FileNotFoundError(2, 'No such file or directory')
                summary: Internal error occurs during execution of update process .

From above logs observed issue is related to pyVmomi. To validate python module on problematic vCenter use command below.

# python -V
Python 3.7.5

# echo $PYTHONPATH

# echo $VMWARE_PYTHON_PATH
/usr/lib/vmware/site-packages

# python -c 'import pyVmomi; print(pyVmomi.__file__)'
/usr/lib/python3.7/site-packages/pyvmomi-8.0.2.0.1-py3.7.egg/pyVmomi/__init__.py

# PYTHONPATH=$VMWARE_PYTHON_PATH:$PYTHONPATH python -c 'import pyVmomi; print(pyVmomi.__file__)'
/usr/lib/vmware/site-packages/pyVmomi/__init__.py

The above error here points that an incorrect public version of pyVmomi has been installed and that needs to be re-installed with private version:

ImportError: cannot import name 'lookup' from 'pyVmomi' (/usr/lib/python3.7/site-packages/pyvmomi-8.0.2.0.1-py3.7.egg/pyVmomi/__init__.py)

Environment

vCenter Server

Cause

The default location for pyVmomi is /usr/lib/vmware/site-packages/pyVmomi on vCenter Server. It is not supported to change/modify/install the default python library on vCenter.

Resolution

To resolve this issue, please restore the vCenter from backup. For reference see: Overview of Backup and Restore Options in vCenter Server 6.x/7.0.x/8.0.