cmsso-util command fails with "ERROR_LOGIN_FAILURE" and "faultcode: ns0:FailedAuthentication" despite correct login credentials

Products

VMware vCenter Server

Issue/Introduction

When you copy `cmsso-util` commands from Windows Notepad to vCenter Server, you may experience the following symptoms:

Authentication fails with "Invalid credentials" despite using the correct password
The error `dir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326)` appears
The username parameter shows corruption with invisible Unicode characters (e.g., `\ufeffadministrator` instead of `administrator`)
Single Sign-On (SSO) domain operations cannot be completed

Command Line Failure

When this issue occurs, you see authentication failures despite entering the correct password:

root@vcenter [ ~ ]# cmsso-util unregister --node-pnid <node-fqdn> --username [email protected] --passwd <password>
YYYY-MM-DDTHH:MM:SS.SSSZ   Running command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', '\[email protected]']
YYYY-MM-DDTHH:MM:SS.SSSZ   Done running command
YYYY-MM-DDTHH:MM:SS.SSSZ   Invoked command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', '\[email protected]']
YYYY-MM-DDTHH:MM:SS.SSSZ   RC = 40
stdout = Enter password for [email protected]:

stderr = dir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326)

Failed. Details {
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "Command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', '\\[email protected]']\nStderr: dir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326) \n"
            ]
        }
    ]
}

Note the corrupted username \ufeffadministrator instead of administrator.

SOAP Authentication Errors

The unregister operation fails with authentication errors:

Failed to unregister service <service-id>. Details SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Invalid credentials
faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

Log File Errors

In /var/log/vmware/vmdird/vmdird.log:

YYYY-MM-DDTHH:MM:SS.SSSZ:t@############:ERROR: VmDirSRPGetIdentityData ([email protected]) failed, (9106)
YYYY-MM-DDTHH:MM:SS.SSSZ:t@############:ERROR: SASLSessionStart: sasl error (-20)(SASL(-13): user not found: no secret in database)
YYYY-MM-DDTHH:MM:SS.SSSZ:t@############:ERROR: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "", Method: SASL

Environment

vCenter Server 7.0 and later, Enhanced Linked Mode
Windows workstation using Notepad with UTF-8 with BOM encoding

Cause

Windows Notepad inserts an invisible Unicode Byte Order Mark (BOM) character (U+FEFF) when saving or copying text using "UTF-8 with BOM" encoding. When you copy commands from Notepad and paste them into vCenter Server, the BOM character is inserted at the beginning of the pasted text.

The BOM character becomes part of the first parameter in the command. In `cmsso-util` commands, this corrupts the administrator username parameter, changing `[email protected]` to `\[email protected]`.

The Linux-based vCenter Server authentication system does not recognize usernames containing BOM characters. Even though the visible text appears correct in the terminal, the invisible BOM character causes the authentication system to reject the credentials as invalid, resulting in ERROR_LOGON_FAILURE (1326).

Resolution

Prevent Windows Notepad from inserting Unicode characters into your commands.

Option 1: Type Commands Manually

Open an SSH session to the vCenter Server Appliance.
Type the cmsso-util command directly in the terminal.
Press Enter to execute the command.

Option 2: Use Correct Encoding

Type your command in Notepad.
Click File > Save As.
Select "ANSI" or "UTF-8" from the Encoding dropdown.
Avoid "UTF-8 with BOM" as this adds the problematic character.
Save the file.
Reopen the file and copy the command.
Paste into the vCenter Server Appliance terminal.

Option 3: Use UTF-16 Encoding

Type your command in Notepad.
Click File > Save As.
Select "UTF-16 BE" or "UTF-16 LE" from the Encoding dropdown.
Save the file.
Reopen the file and copy the command.
Paste into the vCenter Server Appliance terminal.

Additional Information

Affected Commands

These vCenter Server commands can experience this issue when copied from Windows Notepad:

cmsso-util domain-repoint
cmsso-util unregister
dir-cli service list
vdcrepadmin operations
ldapmodify commands

Log Locations

The Unicode BOM character appears in these logs:

/var/log/vmware/cloudvm/cmsso_util.log - Shows \ufeffadministrator
/var/log/vmware/vmdird/vmdird.log - Shows "user not found: no secret in database"
/var/log/vmware/sso/ssoAdminServer.log - Shows authentication failure entries

Alternative Text Editors

Consider using text editors that handle cross-platform text encoding correctly. These editors avoid the BOM character issue when working with vCenter commands.