Patching an ELM vCenter 7.x fails with 'Error 46 while creating SSO group \"NsxAdministrators\":\ndir-cli failed. Error 1326: Operation failed with error ERROR_LOGON

Products

VMware vCenter Server

Issue/Introduction

Patching the vCenter Server fails with the following error :

/var/log/vmware/applmgmt/patchrunnner.log logs reflects below trace

[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 213, in doPatching
    doIncrementalPatching(current_version)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 340, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: {
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "Error 46 while creating SSO group \"NsxAdministrators\":\ndir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326) \n"
            ],
            "localized": "An error occurred while invoking external command : 'Error 46 while creating SSO group \"NsxAdministrators\":\ndir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326) \n'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}.
[YYYY-MM-DDTHH:MM:SS] ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
    reportQueue, identifier, expectedResultType)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/seat/software-updatettcq1_rn/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
[YYYY-MM-DDTHH:MM:SS] WARNING root stopping status aggregation...
[YYYY-MM-DDTHH:MM:SS] ERROR __main__ Patch vCSA failed

Environment

vCenter Server Appliance 7.x
vCenter Server Appliance 8.x

Cause

Due to Minimum Length exceeding 20 Characters, the machine account password on vCenter will have generated greater than 20 Characters and this would have taken place for all vCenter's in Linked mode.

HKEY_THIS_MACHINE\services\vmdir> ls

[\services\vmdir\]
+ "Arguments" REG_SZ "/usr/lib/vmware-vmdir/sbin/vmdird -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -L /var/log/vmware/vmdird/vmdird.log"
+ "Autostart" REG_DWORD 0x00000001 (1)
+ "dcAccount" REG_SZ "vcenter.sample.local"
+ "dcAccountDN" REG_SZ "cn=vcenter.sample.local,ou=Domain Controllers,dc=vsphere,dc=local"
+ "dcAccountOldPassword" REG_SZ "####################"
+ "dcAccountPassword" REG_SZ "########################################"

vCenter SSO Password Policy - Edit the vCenter Single Sign-On Password Policy

As per document:-
The password policy picks up the maximum length value only if the minimum length is greater than 20 characters. The behavior of the password policy is undefined or could result in failure of services when the minimum length value is greater than 20 characters and the maximum length is set to any value. To avoid a potential problem, leave the minimum length set to the default value of 8 characters, or no greater than 20 characters.

Resolution

Step-1

Edit the vCenter Single Sign-On Password Policy as per the following Edit the vCenter Single Sign-On Password Policy.
Note: At Step 5 Click Edit for the Password Policy row.
Set the "Minimum length" to 20(irrespective of whatever value is set in "Maximum length" field).We can leave the "Minimum length" to default 8 or below 20 so that it does not generate a machine account password as per "Maximum length"

Verify the Machine Account Password with below commands and make sure it reflect 20 characters or below and if the Machine Account Password is above 20 Characters then follow through the Step-2 below.

Putty to vCenter

Switch to the appliance shell
Ensure that the lwregshell file has execute permissions. You can check this by running:
- ls -l /opt/likewise/bin/lwregshell
The output will show permissions in the format -rwxr-xr-x or similar. If it doesn’t have execute permissions, you can add them with:
- sudo chmod +x /opt/likewise/bin/lwregshell
To execute lwregshell, type:
- /opt/likewise/bin/lwregshell
To check the password type:
- cd HKEY_THIS_MACHINE\Services\vmdir\
The output should reflect similar to below, highlighted in green.

HKEY_THIS_MACHINE\services\vmdir> ls

[\services\vmdir\]
+ "Arguments" REG_SZ "/usr/lib/vmware-vmdir/sbin/vmdird -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -L /var/log/vmware/vmdird/vmdird.log"
+ "Autostart" REG_DWORD 0x00000001 (1)
+ "dcAccount" REG_SZ "vcenter.sample.local"
+ "dcAccountDN" REG_SZ "cn=vcenter.sample.local,ou=Domain Controllers,dc=vsphere,dc=local"
+ "dcAccountOldPassword" REG_SZ "####################"
+ "dcAccountPassword" REG_SZ "########################################"

Step-2 (Once the Password Policy is changed, Reset the Machine Account Password on all vCenter in Enhanced Linked Mode. )

Copy the script attached to this article on the vCenter Server or PSC which is facing the issue with Invalid Credentials

For Manually resetting Machine Account Password refer to Steps in KB- "LDAP Error Code 49"/Error (49) error in vmdird logs in vCenter Server

Verify that the vmdir database is in normal state:

/usr/lib/vmware-vmafd/bin/dir-cli state get

The output should look like this:

Directory Server State: Normal (3)

If the vmdir database is not in normal state, change it by running:

/usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL

Make the script executable by executing the following command:

chmod +x reset_machine_pw.sh
Download the script from KB- LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script

Run the script. You will be prompted for the administrator@<sso.domain> password and replication partner name if it is executed on PSC or Embedded Node

Restart all the services after modifying the password:
service-control --stop --all && service-control --start --all

Once all the vCenter in Linked mode reflets the Machine Account Password with 20 characters or below, proceed to upgrade the vCenter.

Note: Make sure that there is no replication issues between the Linked mode vCenter's.