Workload Domain Deployment in SDDC Manager 5.1.x Fails with Error "VMDIR Service Is Not in 'NORMAL' State".

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:

 

• In the SDDC Manager UI, the task to join the vCenter Server to the Single Sign-On (SSO) Ring Topology shows the following error:
  
  "Failed to join new SSO node vcenter.domain.com

  Cause: Retriable operation 'Waiting to get the state of VMDIR service for vcenter.domain.com' failed to complete after 10 retries. VMDIR service is not in 'NORMAL' state."

• SDDC manager logs contains errors similar to the excerpt below:
  
  /var/log/vmware/vcf/domainmanager/domainmanager.log

  DEBUG [vcf_dm,668e96c2cd759f46e205153f80dbc722,b0c3] [c.v.evo.sddc.common.util.SshUtil,dm-exec-8]  The command [ /usr/lib/vmware-vmafd/bin/dir-cli state get --password **************** ] executed on vcenter.domain.com. Status: 246, Timed out: false
  Output:
  Error: dir-cli failed. Error 382312694: Access denied, reason = rpc_s_auth_method (0x16c9a0f6).
  ..
  ERROR [vcf_dm,655dbbe9d9225aea48d1e9a3e091c0b0,8153] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-9]  [OHTOJR]
  FAILED_TO_JOIN_SSO_NODE Failed to join new SSO node vcenter.domain.com
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to join new SSO node vcenter.domain.com
  ..
  Caused by: java.lang.RuntimeException: Retriable operation 'Waiting to get the state of VMDIR service for vcenter.domain.com' failed to complete after 10 retries

• vCenter logs contains errors similar to the excerpt below:
  
  /var/log/vmware/vmdird/vmdird-syslog.log

  t@139923633980992:ERROR: VdirPasswordFailEvent from user(cn=administrator,cn=users,dc=vsphere,dc=local), error(0)()
  t@139923633980992:ERROR: Srv_rpc_srp_verifier_verify_session failed, status (382312692)

• The VMDIR of the vCenter is confirmed to be in NORMAL state.
  
  Note: You can verify by executing the following command on the vCenter in BASH mode:
  
  

  # /usr/lib/vmware-vmafd/bin/dir-cli state get

  
  
  
  
• The SSO administrator password used by SDDC Manager is confirmed to be functional.

 

Environment

VMware Cloud foundation 5.1.x

Cause

The issue occurs because the SSO password is misinterpreted during workflow execution due to the presence of special characters.

This problem affects only VCF version 5.1.x.

Resolution

• Fixed in VCF version 5.2.

   

  Workaround:

  1. Delete the failed workload domain: Delete a VI Workload Domain.
  2. Change the SSO administrator account password from SDDC Manager Password Management to a password that does not contain @, \, =, #, $, &, ", parentheses, square or curly brackets.
     • Password for [email protected] cannot be update/rotate/remediate from SDDC Manager when logged in as [email protected]. use another account with SSO admin privileges and has admin access to SDDC Manager
  3. Reattempt the deployment.

   

  Note: If deletion of the failed workload domain is not possible open a case with VCF Global Support for assistance.