vCenter patching fails with error "ERROR_OPEN_FAILED" while updating solution users in vmdir
search cancel

vCenter patching fails with error "ERROR_OPEN_FAILED" while updating solution users in vmdir

book

Article ID: 324592

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • This issue can occur when patching is done via VAMI or CLI
  • In the past, vmca-root certificate was replaced on the affected vCenter using an API(http://localhost/api/vcenter/certificate-management/vcenter/vmca-root)
  • In the var/log/vmware/applmgmt/PatchRunner.log, you may find entries similar to:

[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR root Failed to update solution user wcp.
Traceback (most recent call last):
  File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 314, in update
    self.service.update(self._soluser_id, self._cert_path)
  File "/usr/lib/vmware/site-packages/cis/vecs.py", line 304, in update
    raise ex
  File "/usr/lib/vmware/site-packages/cis/vecs.py", line 298, in update
    invoke_command(cmd, encoding=sys.getdefaultencoding())
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 369, in invoke_command
    (cmd, stderr))
cis.exceptions.InvokeCommandException: {
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "Command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'update', '--name', 'wcp-058f48f7-826e-4a71-acf0-09420bba7a46', '--cert', '/tmp/wcp_2nb1qa9a.crt', '--login'
, 'vc.vmware.com', '--password', 'XXXXX']\nStderr: dir-cli failed. Error 110: Operation failed with error ERROR_OPEN_FAILED (110) \n"
            ],
            "localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'update', '--name', 'wcp-058f48f7-826e-4a71-acf0-09420bba7a46', '--cert', '/tmp/wcp_2nb1qa9a.crt', '--login', 'vc.vmware.com', '--password', 'XXXXX']\nStderr: dir-cli failed. Error 110: Operation failed with error ERROR_OPEN_FAILED (110) \n'"
        },
        {
            "id": "install.ciscommon.vecs.update.entry",
            "translatable": "Error in updating service entry %(0)s using certificate %(1)s.",
            "args": [
                "wcp-058f48f7-826e-4a71-acf0-09420bba7a46",
                "/tmp/wcp_2nb1qa9a.crt"
            ],
            "localized": "Error in updating service entry wcp-058f48f7-826e-4a71-acf0-09420bba7a46 using certificate /tmp/wcp_2nb1qa9a.crt."
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

[YYYY-MM-DDTHH:MM:SS] ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/archive/software-updaten1a0jnzx/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/archive/software-updaten1a0jnzx/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/archive/software-updaten1a0jnzx/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
    reportQueue, identifier, expectedResultType)
  File "/storage/archive/software-updaten1a0jnzx/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/archive/software-updaten1a0jnzx/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
[YYYY-MM-DDTHH:MM:SS] WARNING root stopping status aggregation...
[YYYY-MM-DDTHH:MM:SS] ERROR __main__ Patch vCSA failed

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

  • This issue is caused when the solution user certificates in vmdir are in a unsupported format.
  • When the vmca-root certificate is replaced using API, the solution user certificates in vmdir are double encoded.
  • During vCenter patching, when the workflow tries to update the solution user certificate in vmdir, it will fail with error "ERROR_OPEN_FAILED"

Additional information:

  • When the solution user certificates in vmdir are in a unsupported format, any operation that involves updating solution users using dir-cli service update command will fail with this error.

Some of the operations that may fail are:

  • Changing vCenter PNID
  • Resetting all the vCenter Certificates or solution user certificates

Resolution

This is a known issue affecting vcenter versions before 7.0U3O and 8.0b The issue with certificate replacement API is fixed in vCenter Server 7.0 Update 3O and vCenter Server 8.0b

Note: Patching vCenter to the above versions is not the solution for this issue. From these versions, if the API is used to replace the vmca-root certificate, the solution user certificates in vmdir are formatted correctly and this issue will not occur during the next patching attempts.

Workaround:
Reviewing the solution user certificates in vmdir

  • To identify if the solution user certificates in vmdir are in a correct format or not, use any of the below tools:

VCF Diagnostic Tool :

  • Run vdt 2.0.x on vCenter which has "Solution User Cert Parity Checks". This check will compare the solution user certificates between VECS and VMDIR and report inconsistencies.

Example Output from vdt for this issue:

[YYYY-MM-DDTHH:MM:SS] ERROR Solution User Cert Parity Checks get_soluser_cert: Unable to load certificate
[YYYY-MM-DDTHH:MM:SS] ERROR Solution User Cert Parity Checks get_soluser_cert: Unable to load certificate
[YYYY-MM-DDTHH:MM:SS] ERROR Solution User Cert Parity Checks get_soluser_cert: Unable to load certificate
[YYYY-MM-DDTHH:MM:SS] ERROR Solution User Cert Parity Checks get_soluser_cert: Unable to load certificate
[YYYY-MM-DDTHH:MM:SS] ERROR Solution User Cert Parity Checks get_soluser_cert: Unable to load certificate
[FAIL]    Solution User Cert Parity Checks

Reference: Using the VCF Diagnostic Tool for vSphere (VDT)

vCert :

  • Run vCert( Option 1 Check current certificates status) which also has a check to validate the solution user certificates.

Example output from vCert for this issue:

Checking Certificate Status
-----------------------------------------------------------------
Checking Machine SSL certificate                            VALID
Checking Solution User certificates:
   machine                                               MISMATCH
   vsphere-webclient                                     MISMATCH
   vpxd                                                  MISMATCH
   vpxd-extension                                        MISMATCH
   hvc                                                   MISMATCH
   wcp                                                      VALID

Reference :  vCert - Scripted vCenter Expired Certificate Replacement

Note: Even though the wcp solution user certificate shows valid in the above results, updating the  wcp solution user certificate using dir-cli will still fail.

Resolution:

  • To resolve this issue, we need to recreate the solution users in vmdir with correct certificates.
  • This can be done manually or using lsdoctor (lsdoctor -u)

Reference: Using the 'lsdoctor' Tool 

Note: For this specific issue, we may need to run the lsdoctor -u twice if all the solution users are not updated in the first attempt.

Powered by Wolken Software