WCP service fails to start with Error 46 while creating SSO group
search cancel

WCP service fails to start with Error 46 while creating SSO group

book

Article ID: 379776

calendar_today

Updated On:

Products

VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • WCP (Workload Control Plane) service fails to start 
  • As the WCP service is in stopped state, administrator cannot set any ESXi host in maintenance mode using vCenter Server
  • Upon starting the wcp services it fails with the following error 
    root@<FQDN of VC>#service-control --start wcp

Operation not cancellable. Please wait for it to finish...
Performing start operation on service wcp...

stderr: Error executing start on service wcp. Details {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"wcp"
],
"localized": "An error occurred while starting service 'wcp'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
  • From /var/log/vmware/vmon/vmon.log
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03) host-2515 <wcp> Service pre-start command's stderr: Failed to configure HDCS. Err {hh:mm:ss.X
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515     "detail": [
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515         {
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515             "id": "install.ciscommon.command.errinvoke",
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515             "translatable": "An error occurred while invoking external command : '%(0)s'",
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515             "args": [
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515                 "Error 46 while finding SSO group \"vCLSAdmin\":\ndir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326) \n"
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515             ],
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515             "localized": "An error occurred while invoking external command : 'Error 46 while finding SSO group \"vCLSAdmin\":\ndir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326) \n'"
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515         }
YYYY-MM-DDTXXZ Wa(03)+ host-2515     ],
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515     "componentKey": null,
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515     "problemId": null,
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515     "resolution": null
YYYY-MM-DDThh:mm:ss.XXXZ Wa(03)+ host-2515 }
YYYY-MM-DDThh:mm:ss.XXXZ Er(02) host-2515 <wcp> Service pre-start command failed with exit code 1.
  • From /var/log/vmware/vmdird/vmdird-syslog.log

 

YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055553222400: _VmDirCpMdbFile: completed making snapshot with file size 32Mb in 1 seconds; data transfer rate: 32.0MB/sec, db last tid: 24589
YYYY-MM-DDThh:mm:ss.XXXZ err vmdird  t@140054890526464: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
YYYY-MM-DDThh:mm:ss.XXXZ err vmdird  t@140054890526464: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
YYYY-MM-DDThh:mm:ss.XXXZ err vmdird  t@140054890526464: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "cn=<FQDN of VCENTER>,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL
YYYY-MM-DDT2hh:mm:ss.XXXZ info vmdird  t@140055427397376: MOD 1,rep,certificateRevocationList: (-----BEGIN X509 CRL-----
MIICIzCCAQsCAQEwDQYJKoZIhvcNAQELBQAwgaYxCzAJBgNVBAMMAkNBMRcwFQYK
CZImiZPyLGQBXXXyyyVwaGVyZTEWMBQGCgmSJomT8ixkARkWBmNsc3BwZDELMAkG
A1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaYUihJzAlBgNVBAoMHmNsc3BwZHZj
c3AwMS5jbHNwcmVwcm9kLmlibWNsczEbMBkG)
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055427397376: Modify Entry (CN=E91E8F0B4C40A221275985C73F827BB58315476E,CN=Certificate-Authorities,cn=Configuration,dc=vsphere,dc=local, EID 2123)(from 127.0.0.1)(by <FQDN of VCENTER>@vsphere.local)(via Ext)(USN 12318,0)
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055553222400: _VmDirCpMdbFile: making database snapshot with file size 32Mb; will take approximate 1 seconds; 1 updates occurred since last snapshot.
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055553222400: _VmDirCpMdbFile: completed making snapshot with file size 32Mb in 1 seconds; data transfer rate: 32.0MB/sec, db last tid: 24591
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055721010944: VmDirGetAccountUPN success for AccountUPN (workload_storage_management-27789762-bca9-434f-810a-8c83b91b914b@VSPHERE.local)
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055721010944: Srv_RpcVmDirGetAccountUPN success AccountUPN Length (79)
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055419004672: Modify Entry (CN=workload_storage_management-27789762-bca9-434f-810a-8c83b91b914b,cn=ServicePrincipals,dc=vsphere,dc=local, EID 3237)(from )(by )(via Int)(USN 12319,0)
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055419004672: Modify Entry (CN=workload_storage_management-27789762-bca9-434f-810a-8c83b91b914b,cn=ServicePrincipals,dc=vsphere,dc=local, EID 3237)(from )(by )(via Int)(USN 12320,0)
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055419004672: User account control - (cn=workload_storage_management-27789762-bca9-434f-810a-8c83b91b914b,cn=serviceprincipals,dc=vsphere,dc=local): (800010) flag unset, new value=(0)
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055419004672: Password Modification Successful (). Bind DN: "". Modified DN: "CN=workload_storage_management-27789762-bca9-434f-810a-8c83b91b914b,cn=ServicePrincipals,dc=vsphere,dc=local"
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055419004672: VmDirSrvForceResetPassword (workload_storage_management-27789762-bca9-434f-810a-8c83b91b914b@VSPHERE.local)
YYYY-MM-DDhh:mm:ss.XXXZ info vmdird  t@140055553222400: _VmDirCpMdbFile: making database snapshot with file size 32Mb; will take approximate 1 seconds; 2 updates occurred since last snapshot.
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055553222400: _VmDirCpMdbFile: completed making snapshot with file size 32Mb in 1 seconds; data transfer rate: 32.0MB/sec, db last tid: 24595
YYYY-MM-DDThh:mm:ss.XXXZ info vmdird  t@140055427397376: MOD 1,rep,certificateRevocationList: (-----BEGIN X509 CRL-----

 

Environment

VMware vCenter Server Appliance 7.0.x
VMware vCenter Server Appliance 8.0.x

Cause

The issue is only seen if the machine account password is beyond 20 characters which can be tested by setting the "vmwPasswordMinLength" to above 20.

Resolution

Take offline (powered off) snapshots of all PSC's and VC's in the same vSphere Domain (or in ELM) before attempting.  This is standard best practice before making any manual changes to the PSC VMDIRD database.


To resolve the issue, follow below mentioned steps 



    • Connect to VCSA via SSH by using root credentials 

    • Type "shell" to gain access in shell mode 

    • To verify number of characters in "dcAccountPassword" run below mentioned command


    /opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\services\vmdir]' | egrep -i "Password|dcAccountDN"



    • If the output of the above-mentioned command looks like,


+  "dcAccountDN"          REG_SZ          "cn=<FQDN OF vCenter>=Domain Controllers,dc=example,dc=local"
+  "dcAccountOldPassword" REG_SZ          "`<XXXXXXXXXXXXXXXXXXX>"
+  "dcAccountPassword"    REG_SZ          <XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX>


    • If the "dcAccountPassword" is more that 20 characters then the machine account password needs to be changed

    • To reset the machine account password, use below mentioned KB article.


        LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script



    • Once the machine account is reset, restart all the service of the vCenter server


service-control --stop --all && service-control --start --all

 


NOTE:  You may receive an error when you try to run the script:


bash:  ./reset_machine_pw.sh: /bin/bash^M: bad interpreter: No such file or directory

This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor.  To resolve this problem:


    • run the following command:
      # sed -i -e 's/\r$//' reset_machine_pw.sh

    • Rerun the script.

Additional Information

About vSphere Authentication

For VMware-vSphere 7.X 

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-B9C4409A-B053-40C3-96DE-232BB99AAA35.html

For VMware-vSphere 8.X 

https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-authentication/GUID-43527B09-63BB-44A6-91D3-E3A470904698.html

"Note: The password policy picks up the maximum length value only if the minimum length is greater than 20 characters. The behavior of the password policy is undefined or could result in failure of services when the minimum length value is greater than 20 characters and the maximum length is set to any value. To avoid a potential problem, leave the minimum length set to the default value of 8 characters, or no greater than 20 characters."

Powered by Wolken Software