Repointing to new SSO domain fails starting vpxd-svcs when SSO and AD domain are the same.
search cancel

Repointing to new SSO domain fails starting vpxd-svcs when SSO and AD domain are the same.

book

Article ID: 319433

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article provides steps to identify the cause for failure encountered while adding AD over LDAP if the AD has the same domain name as the SSO domain. Or issue encountered while repointing to new SSO domain if current SSO domain name is the same as AD domain. In both instances, vpxd-svcs service fails to start.

During the SSO domain repoint vpxd-svcs service fails with the below log entries in /var/log/vmware/vpxd/vpxd-svcs.log:

YYYY-MM-DDTHH:MM:SS Wa(03) host-##### <vpxd-svcs> Service pre-start command's stderr: YYYY-MM-DDTHH:MM:SS Security error: (vmodl.fault.SecurityError) {
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### dynamicType = <unset>,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### dynamicProperty = (vmodl.DynamicProperty) [],
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### msg = '',
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### faultCause = <unset>,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### faultMessage = (vmodl.LocalizableMessage) []
YYYY-MM-DDTHH:MM:SS Wa(03) host-##### <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 260, in securityctx_modifier
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### return req_method(self, *args, **kargs)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### self.f(*(self.args + (obj,) + args), **kwargs)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### return self._stub.InvokeMethod(self, info, args)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### raise obj # pylint: disable-msg=E0702
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### pyVmomi.VmomiSupport.SecurityError: (vmodl.fault.SecurityError) {
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### dynamicType = <unset>,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### dynamicProperty = (vmodl.DynamicProperty) [],
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### msg = '',
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### faultCause = <unset>,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-##### faultMessage = (vmodl.LocalizableMessage) []

Environment

vCenter Server 7.x
vCenter Server 8.x

Cause

During a domain repoint the service registrations get exported and updated with the new SSO domain name. In some cases this is not always successful. To determine if you are experiencing this issue you can export out the service registrations with lstool.py from the node that failed to repoint (before reverting back)  and look for the old domain name within the service registrations.
  1. Run this command on the vCenter via SSH to get a list of service registrations

    /usr/lib/vmware-lookupsvc/tools/lstool.py list --url 'http://localhost:7090/lookupservice/sdk' > /tmp/services.txt

  2. Run this command to determine if the old SSO domain name is still present within the service registrations

    grep -i wrong.local /tmp/services.txt

In this example we see that the vmwLKUPOwnerId's  are incorrect and still referencing the old SSO domain name WRONG.LOCAL

vmwLKUPOwnerId: machine-########-####-####-####-############@WRONG.LOCAL
vmwLKUPOwnerId: machine-########-####-####-####-############@WRONG.LOCAL
vmwLKUPOwnerId: machine-########-####-####-####-############@WRONG.LOCAL
vmwLKUPOwnerId: machine-########-####-####-####-############@WRONG.LOCAL

Resolution

Please note that, further troubleshooting requires offline snapshot of VCSA. If this VCSA is participating in ELM then offline snapshots of all VCSA in ELM are required.

Resolution Steps:

  1. Set DRS to manual.
  2. Proceed with the domain repoint by running the below command

    cmsso-util domain-repoint -m execute --src-emb-admin Administrator --dest-domain-name <destination_PSC_domain> --debug

    Note Replace the <destination_PSC_domain> with the new domain of the vCenter.

  3. Run the below command

    tail -f /var/log/vmware/cloudvm/cmsso_util.log |grep -i "End Getting Endpoint Specs"
     
  4. Open a new SSH session and run the below command to change the spec files to the correct owner ID and the repoint process will register them back

    cd /storage/domain-data/service-phase-data/specs && sed -i "s|@WRONG.LOCAL|@RIGHT.LOCAL|g" * && sed -i "s|@wrong.local|@right.local|g" *

    NOTE: Make sure to search for the old domain regardless of case sensitivity
    NOTE: In this example, we are changing all instances of "@WRONG.LOCAL" to "@RIGHT.LOCAL"  to fix the vmwLKUPOwnerId's. You may need to modify or change other entries according to what is wrong in your environment.

  5. Proceed by adding the identity source over LDAP.
Powered by Wolken Software