Pre-Check error: "Internal error occurs during execution of update process" when updating vCenter server from 8.0U2b to 8.0U3

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• Updating vCenter Server Appliance (VCSA) fails during precheck with the error "Internal error occurs during execution of upgrade process".

 

• From the var/log/vmware/applmgmt/Patchrunner.log we see following events :

YYYY-MM-DDThh:mm:ss.621Z vmdir:CollectRequirements ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vmdir:CollectRequirements' failed.
Traceback (most recent call last):
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/payload/components-script/vmdir/__init__.py", line 513, in collectRequrements
    repStatusOutput = validator.validate()
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/payload/components-script/vmdir/replication_validator.py", line 36, in validate
    return invoke_command(partnerStatusCommand, stdin=self._ssoUserPassword)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 372, in invoke_command
    raise InvokeCommandException(errStr='Command: %s\nStderr: %s' %\
cis.exceptions.InvokeCommandException: {
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "Command: ['/usr/lib/vmware-vmdir/bin/vdcrepadmin', '-f', 'showpartnerstatus', '-h', 'localhost', '-u', '<vCenter_FQDN>']\nStderr: "
            ],
            "localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmdir/bin/vdcrepadmin', '-f', 'showpartnerstatus', '-h', 'localhost', '-u', '<vCenter_FQDN>']\nStderr: '"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
YYYY-MM-DDThh:mm:ss.624Z ERROR vmware_b2b.patching.phases.discoverer Could not execute discovery hook in file: /storage/core/software-update/updates/8.0.3.00000/scripts/patches/payload/components-script/vmdir
YYYY-MM-DDThh:mm:ss.625Z ERROR vmware_b2b.patching.phases.discoverer Discovery hook got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/phases/discoverer.py", line 580, in discover
    discoveredComponents = _discoverComponents(scriptsRootDir, stageDir, components,
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/phases/discoverer.py", line 214, in _discoverComponents
    requirementsResult = executeHook(filePath, Hook.Requirements, compContext, None,
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
YYYY-MM-DDThh:mm:ss.629Z ERROR __main__ Discovery of vCSA patching components failed

 

• From /usr/lib/vmware-vmdir/bin , manually run the vdcrepadmin , vdcleavefed script ( ./vdcrepadmin , ./vdcleavefed ) and it failed with the below IO error :
  Vdcrepadmin failed. Error [IO error] [9003]

 

• Moved the vdcrepadmin , vdcleavefed script from the impacted vCenter to a working vCenter's /var/core or /tmp directory and tried running. And, it will successfully execute.

Cause

• vmdir directory is missing from var/log/vmware

Resolution

• Access the Appliance via SSH:
        Open your SSH client 
        Connect to the IP address or hostname of your VMware appliance using a privileged account (e.g., root).
• Create the Missing Directory:
          Manually create the required directory /var/log/vmware/vmdir using the mkdir command. 
          This ensures the necessary path exists for the VAMI pre-check and subsequent operations.
                    mkdir -p /var/log/vmware/vmdir
•  Set Directory Permissions:
          After creating the directory, it's crucial to set the correct permissions. 
          chmod 770 /var/log/vmware/vmdir
• Re-run the VAMI Update Pre-check:
          Once the directory is created and its permissions are correctly set, return to the VAMI interface re-initiate the VAMI update pre-check.
• Verify Pre-check Success: The pre-check should now complete successfully without errors related to the missing directory.