"LDAP Error Code 49"/Error (49) error in vmdird logs in vCenter Server
search cancel

"LDAP Error Code 49"/Error (49) error in vmdird logs in vCenter Server

book

Article ID: 319348

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

**Note for vCenter 7.0 and 8.0:**   To determine whether this process is necessary, use the `vmdir_tool.py` utility as described in *Using vmdir_tool.py to identify vmdir/ELM replication inconsistencies*. This process should only be performed if the Bind DN object is the **only** inconsistency reported by the tool. Running it on other objects, such as the 'Domain Controller' account, may cause *vmdir* to become read-only.

 

  • In the vmdird-syslog.log file, the following entries "error 49" may be observed:
[YYYY-MM-DDTHH:MM:SS] err vmdird t@###########: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
[YYYY-MM-DDTHH:MM:SS] err vmdird t@###########: VmDirSendLdapResult: Request (96), Error (49), Message ((49)(SASL step failed.)), (0) socket ([17] x.x.x.x :389<-xxx.xx.xxx.xxx:54753)
[YYYY-MM-DDTHH:MM:SS] err vmdird t@###########: Bind Request Failed ([17] x.x.x.x:389<-x.x.x.x:54753) error 49: Protocol version: 3, Bind DN: "cn=accountname,ou=Computers,dc=vsphere,dc=local", Method: 163
  • For PSC nodes, the Bind DN is displayed in the following format: **Bind DN:** `"cn=FQDN.local,ou=Domain Controllers,dc=vsphere,dc=local"`
[YYYY-MM-DDTHH:MM:SS] err vmdird t@#############: Bind Request Failed ([17] x.x.x.x :389<-x.x.x.x:54753) error 49: Protocol version: 3, Bind DN: "cn=FQDN.local,ou=Computers,dc=vsphere,dc=local", Method: 163


Then FQDN.local would be the affected account.

The SSO domain, which we can also see in the error message, is vsphere.local

Therefore, the entire machine account name in this example will look like: [email protected]

Note: The vmdird-syslog.log file is located at:

For vCenter Server Appliance with embedded Platform Services Controller(PSC): /var/log/vmware/vmdird/vmdird-syslog.log

For Windows installed vCenter Server with embedded Platform Services Controller(PSC): "%VMWARE_LOG_DIR%"\vmdird\vmdir.log

 

Note: The vmdir log is not present in vCenter's that do not have an embedded PSC.

Note: From 6.5 onwards inventory services is not available, For LDAP errors see /var/log/vmware/sso/vmware-sts-idmd.log  or /var/log/vmware/vmdird/vmdird-syslog.log

  • vmdir replication may not be working between vCenter with Embedded PSCs/External PSCs - nodes may be XXX changes behind from replication partners' point of view

Replication can be checked using the  command (must be run on each VC/PSC in the SSO domain to accurately reflect the situation):

# /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

 

If a partner is changes behind, review the vmdird-syslog.log of both nodes for ldap 49 errors against those machines.

Environment

VMware VCenter server

Cause

This issue arises when a machine loses trust due to a password mismatch in *vmdird* for the account referenced in the *vmdird-syslog.log* file.

Such mismatches typically occur if the vCenter Server or PSC has been restored from an older backup or snapshot.

Resolution

To fix this issue, reset the password for the user account mentioned in the *vmdird-syslog.log* file.
 

1) Reset the machine account password manually:

Warning: For vdadmintool correct default settings in the SSO password policies are required, VMware currently does not support to set the maximum password length above 20 characters.
 
 
For vCenter Appliance:
  1. Create a snapshot of vCenter Server and Platform Services Controller. If there are multiple vCenters with embedded PSCs in the SSO domain; ensure all machines in the SSO domain are offline, snapshot them all and if restoring; restore all before powering on any machines to ensure consistency in the case of a revert.

  2. Connect to the vCenter/PSC which has the ldap 49 error in it's vmdird-syslog.log with an SSH session and root credentials.

  3. Run this command to enable access the Bash shell:

    # shell.set –enabled true


  4. Type shell and press Enter.

  5. Run this command to open the vdcadmintool:

    # /usr/lib/vmware-vmdir/bin/vdcadmintool

    The following options are presented:

    ================================
    Please select:
    0. exit
    1. Test LDAP connectivity
    2. Force start replication cycle
    3. Reset account password
    4. Set log level and mask
    5. Set vmdir state
    ================================
     
  6. Select option 3.

  7. Enter the user account listed in the vmdird-syslog.log file.

    Note: This is the machine account in the format FQDN@SSO Domain.

    For example:

    [email protected]

    Note: The tool does not filter out invalid characters from the generated password such as:
    & (ampersand)
    ; (semicolon)
    " (double quotation mark)
    ' (single quotation mark)
    ^ (circumflex)
    \ (backslash)
    % (percentage)

    Option 3 may need to be ran several times until it generates a valid password.
     
  8. Make a note of the new auto-generated password.

  9. Connect to vCenter Server Appliance or PSC with an SSH session and root credentials.

  10. Run this command to enable access the Bash shell:

    # shell.set –enabled true

     
  11. Type shell and press Enter.

  12. Run these commands to update the password:

    # /opt/likewise/bin/lwregshell
# cd HKEY_THIS_MACHINE\services\vmdir\
# set_value dcAccountPassword "new password"
# quit

     

  13. Restart the vCenter Server Appliance services. For more information, see Stopping, starting, or restarting VMware vCenter Server Appliance 6.x services (2109887).

 

For Windows installed vCenter Server:

  1. Create a snapshot of the vCenter Server and Platform Services Controller. If there are multiple vCenters with embedded PSCs in the SSO domain; ensure all machines in the SSO domain are offline, snapshot them all and if restoring; restore all before powering on any machines to ensure consistency in the case of a revert.

  2. Open the elevated command prompt on the vCenter with Embedded PSC

  3. Run this command:

    %VMWARE_CIS_HOME%\vmdird\vdcadmintool.exe

    The following options are presented:

    ================================
    Please select:
    0. exit
    1. Test LDAP connectivity
    2. Force start replication cycle
    3. Reset account password
    4. Set log level and mask
    5. Set vmdir state
    ================================
     
  4. Select option 3.

  5. Enter the user account listed in the vmdir.log file.

    Note: This is the machine account in the format FQDN@SSO Domain.

    For example:

    [email protected]

    Option 3 may need to be ran several times until it generates a valid password.
     
  6. Make note of the generated password.

  7. Connect to the vCenter Server or PSC and open regedit.

    Note: Before making any registry modifications, ensure that there is a current and valid backup of the registry and the virtual machine. For more information on backing up and restoring the registry, see the Microsoft article 136393.

  8. Navigate to HLKM\System\CurrentControlset\Services\VMwareDirectoryService\ location.

  9. Update the password for the key dcAccountPassword.

  10. Save the changes and exit.

  11. Restart the vCenter Server services. For more information, see Stopping, starting, or restarting VMware vCenter Server 6.x services (2109881).

 

2) Reset machine account password using dir-cli (versions 6.5 onwards):

Note: Ensure offline snapshots of all vCenter's and PSCs are in place before running. This means to power off all vCenters and PSCs in the SSO domain, login to the ESXi hosts they're placed on and snapshot them when down. If reverting, revert all machines before powering any on. This is to ensure consistency in the SSO domain.

Appliance:

  1. Login to the machine noted in the vmdird-syslog.log as root user via SSH

  2. Run this command to enable access the Bash shell:

    # shell.set –enabled true
     
  3. Type shell and press Enter.

  4. Run below command where <Platform Services Controller FQDN> is the FQDN of the vCenter with embedded PSC or PSC of the machine with the error 49 in it's vmdird-syslog.log:
# /usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname <Platform Services Controller FQDN> --password '<[email protected] password>'

Windows:
  1. Login to the machine noted in the vmdir.log

  2. Open the elevated command prompt on the vCenter with Embedded PSC.

  3. Run below command where <Platform Services Controller FQDN> is the FQDN of the vCenter with embedded PSC or PSC of the machine with the error 49 in it's vmdir.log:
%VMWARE_CIS_HOME%\vmafdd\dir-cli.exe computer password-reset --login administrator --live-dc-hostname <Platform Services Controller FQDN> --password <[email protected] password>

 

3) Reset machine account password using shell script (Appliance only)

LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script (70756)
 

4) Reset using reset_machine_pw.sh shell script (Built in for version 7.0 only)

  1. Take offline snapshots of all vCenter's in the SSO domain before proceeding. This means to power off all vCenter's in the SSO domain, connect to the ESXi hosts they're placed on and snapshot each of them while in powered off state. If reverting; restore each to snapshot before powering any on. This ensure consistency of the SSO domain.

  2. Connect to the vCenter over SSH with root user and type shell to access the bash shell

  3. Run the script using the command below - the prompt will require the FQDN of the replication partners (vCenter's) where the machine account password needs to be reset and also prompted for SSO admin credentials:
# /usr/lib/vmware-vmdir/vmdir-tool/reset_machine_pw.sh

Additional Information

Powered by Wolken Software