Converge fails with "Error: Operation failed with error ERROR_FILE_NOT_FOUND (2)"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• On UI - Converging a vCenter Server with external Platform Service Controller using UI fails with error: Failed to gather requirements after a progress of 8%.
• On CLI - Converge fails with error as follows in converge.log:

2019-05-08T06:06:17.205Z ERROR converge Failed to get vecs users and permissions. Error: {

    "componentKey": null,
    "resolution": null,
    "problemId": null,
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "args": [

                "Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getkey', '--store', 'KMS_ENCRYPTION', '--alias', 'password-cls1/sv1', '--output', '/root/velma/old_certs/password-cls1/sv1-KMS_ENCRYPTION.key']\nStderr: vecs-cli failed. Error 2: Possible errors: \nLDAP error: Protocol error \nWin Error: Operation failed with error ERROR_FILE_NOT_FOUND (2) \n"

            ],

            "localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getkey', '--store', 'KMS_ENCRYPTION', '--alias', 'password-cls1/sv1', '--output', '/root/velma/old_certs/password-cls1/sv1-KMS_ENCRYPTION.key']\nStderr: vecs-cli failed. Error 2: Possible errors: \nLDAP error: Protocol error \nWin Error: Operation failed with error ERROR_FILE_NOT_FOUND (2) \n'",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

Environment

VMware vSphere 6.7.x

Cause

This issue occurs if alias of cert or key of any store in VECS contain a "/"(forward slash).

Resolution

This is a known issue affecting VMware vSphere 6.5 and 6.7 versions.
 

This issue is resolved in vCenter Server 6.7 Update 3, available at Customer Connect. For more information, see the  VMware vCenter Server 6.7 Update 3 Release Notes

 

Workaround:
To work around this issue:

1. Backup the complete store where alias of any cert or key contain slash “/” (KMS_ENCRYPTION as an example only)

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store KMS_ENCRYPTION --alias 'alias-with-slash' --output 'key-with-any-name'

2. Delete all the entries in that store of which we took backup.

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store KMS_ENCRYPTION --alias 'alias-with-slash'

3. Run Converge.
4. Restore the entries as it is in the respective store from which we deleted the entries.

5. Check for entries of a store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store <store-name>

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store KMS_ENCRYPTION –-alias 'alias-with-slash' --key 'key-with-any-name'

Note: If KMS entry is modified, reconfigure the KMS using the following Key Management Server status reports "Not Connected" after convergence to embedded Platform Services Controller.

If all entries of KMS are intact and only the entry with "/" is missing:

1. Log in to https://<VC-IP>/ui.
2. Go to VC > Configure > Key Management Servers and select the KMS.
3. Go to Actions > Edit > Re-enter the password.
4. Click Save.

Additional Information

Impact/Risks:
When there is any network latency, you may see message "Cannot retrieve the required certificate".