Creation of new Workload Domain (WLD) fails at stage "Set SDDC Deployment Details on vCenter Server"
Article ID: 451725
Updated On:
Products
Issue/Introduction
- "Set SDDC Deployment Details on vCenter Server" stage fails while creating a new WLD in SDDC Manager
- Below snippets will be present in SDDC Manager:
/var/log/vmware/vcf/domainmanager/domainmanager.log :
YYYY-MM-DDTHH:MM:SS.160+0000 ERROR [vcf_dm,6a54cbec2e0e90922f1####f4a80,e0ab] [c.v.e.s.v.c.ConfigureDeploymentDetails,dm-exec-14] Could not find deployment setting with key config.SDDC.Deployed.InstanceId in management vCenter VCENTER-FQDN.YYYY-MM-DDTHH:MM:SS.166+0000 ERROR [vcf_dm,6a54cbec2e0e90922f1####f4a80,e0ab] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-14] [5UBDI4] VCF_VSPHERE_CONFIGURE_DEPLOYMENT_DETAILS_FAILED Configuration of deployment details failed.com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Configuration of deployment details failed. at com.vmware.evo.sddc.vsphere.contract.ConfigureDeploymentDetails.execute(ConfigureDeploymentDetails.java:87) at com.vmware.evo.sddc.vsphere.contract.ConfigureDeploymentDetails.execute(ConfigureDeploymentDetails.java:41) at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66) at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161) at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147) at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:403) at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:517) at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128) at jdk.internal.reflect.GeneratedMethodAccessor877.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:569) at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85) at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71) at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Configuration of deployment details failed.
- Login from SDDC Manager to vCenter server fails due as vCenter server rejects the Service User Account credentials:
/var/log/vmware/vcf/domainmanager/domainmanager.log :
YYY-MM-DDTHH:MM:SS.925+0000 ERROR [vcf_dm,6a61b00a236012ab5285######09baf9,ab56] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-4] Cannot complete login due to incorrect credentials: VCENTER-FQDN, svc-####-sddc-####-vc-####@vsphere.local.YYY-MM-DDTHH:MM:SS.095+0000 ERROR [vcf_dm,6a61b00e90a4c1102c######7515a1,bf50] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-17] Cannot complete login due to incorrect credentials: VCENTER-FQDN, svc-####-sddc-####-vc-####@vsphere.local.YYY-MM-DDTHH:MM:SS.207+0000 ERROR [vcf_dm,6a61b0113e5738f31e######c68f1c2,0adc] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-10] Cannot complete login due to incorrect credentials: VCENTER-FQDN, svc-####-sddc-####-vc-####@vsphere.local.YYY-MM-DDTHH:MM:SS.324+0000 ERROR [vcf_dm,6a61b014891a7c0983e######aadf69,6984] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-10] Cannot complete login due to incorrect credentials: VCENTER-FQDN, svc-####-sddc-####-vc-####@vsphere.local.
- vCenter server vmdird logs report an Error : Bind request failed with LDAP Error 49
/var/log/vmware/vmdird/vmdird.log:YYY-MM-DDTHH:MM:SS.012Z:t@14024999277####:ERROR: VdirPasswordFailEvent from user(cn=svc-####-sddc-####-vcYYY-MM-DDTHH:MM:SS.012Z:t@14024999277####:ERROR: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "CN=svc-####-sddc-####-vcYYY-MM-DDTHH:MM:SS.281Z:t@14025070579####:ERROR: VdirPasswordFailEvent from user(cn=svc-####-sddc-####-vc
Environment
VCF 9.0.2
SDDC Manager 9.0.2
vCenter Server 9.0.2
Cause
Affected service account 'svc-####-sddc-####-vc-####' is repeatedly authenticating using a wrong credential.
Resolution
Take SDDC Manager and vCenter server snapshots/backups before proceeding with below steps.
1. Reset the service account credentials in vCenter server: (Provide [email protected] password when asked)
# /usr/lib/vmware-vmafd/bin/dir-cli user modify --account <svc-acct> --password-never-expires
2. Reset the vCenter service account password by running the below command in vCenter server ( replacing the VC FQDN and Credentials in use): [Note down the new password once generated]
# session=$(curl -ks https://VCENTER-FQDN/api/session -u '[email protected]:PASSWORD' -X POST | jq -r .)
# curl -ks 'https://VCENTER-FQDN/api/vcenter/svc-account-mgmt/password?action=reset' -H "vmware-api-session-id: $session" -H 'Content-Type: application/json' -d '{ "account_name": "svc-####-sddc-####-vc-####" }' | jq
3. Remediate the VC SVC user account "svc-####-sddc-####-vc-####" password in SDDC Manager using API Remediate expired Passwords [NOTE: Password remediation may fail using SDDC UI due to character restrictions]
Please reach out to Broadcom Support if you need any assistance with the above API.
4. Test the password reset functionality by executing any of the vCenter server connectivity related task in SDDC Manager e.g. Test CSR generation for the vCenter server from SDDC Manager.
5. Now REDEPLOY the failed Workload Domain in SDDC Manager. [NOTE: RETRY attempt may fail here as SDDC Manager caches the old vCenter service account credentials at the start of the workflow execution]
Feedback
