Failed to register SRM: Access to perform the operation was denied
search cancel

Failed to register SRM: Access to perform the operation was denied

book

Article ID: 382188

calendar_today

Updated On:

Products

VMware vCenter Server VMware Live Recovery

Issue/Introduction

Symptoms:

Registering or Reconfiguring Site Recovery Manger (SRM) fails with the errors. 

ERROR
Operation Failed
Access to perform the operation was denied.
Operation ID: 1c92fc5d-####-4a37-####-fba99128a2a6
10/17/24, 10:03:09 AM -0500

/var/log/vmware/dr/drconfig.log:

2024-11-07T22:24:05.648Z info drconfig[01075] [SRM@6876 sub=LocalRegistrationManager opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] Registering Extension with id 'com.vmware.vcDr' using '/opt/vmware/srm/conf/extension.xml'
2024-11-07T22:24:05.649Z warning drconfig[01075] [SRM@6876 sub=Default opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] Extension message files do not cover the following locales present on VC: ["en_US", "zh-CN", "zh-TW"]
2024-11-07T22:24:06.115Z warning drconfig[01075] [SRM@6876 sub=vmomi.soapStub[413] opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007f5cbc03bda0, h:31, <TCP '10.#.#.# : 53948'>, <TC
P '10.#.#.# : 443'>>), /sdk>, method: updateExtension; code: 500(Internal Server Error)
2024-11-07T22:24:06.116Z error drconfig[01075] [SRM@6876 sub=LocalRegistrationManager opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] MethodFault error: Fault cause: drConfig.fault.HostUnreachableFault
-->
2024-11-07T22:24:06.116Z warning drconfig[01075] [SRM@6876 sub=LocalRegistrationManager opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] Error while registering extension:
--> (drConfig.fault.HostUnreachableFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    address = "vrms.vmware.com",
-->    port = "443",
-->    errorDescription = "Fault cause: vmodl.fault.SystemError
--> "
-->    msg = ""
--> }
--> 
2024-11-07T22:24:06.124Z warning drconfig[01075] [SRM@6876 sub=LocalRegistrationManager opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] Registering Extension with id 'com.vmware.vcDr' again
2024-11-07T22:24:06.125Z verbose drconfig[01255] [SRM@6876 sub=IO.Connection opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] Attempting connection; <resolver p:0x00007f5cb03693e0, 'vrms.vmware.com:443', next:<TCP '10.#.#.# : 443'>>, last e: 0(Success)
2024-11-07T22:24:06.181Z warning drconfig[01075] [SRM@6876 sub=Default opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] Extension message files do not cover the following locales present on VC: ["en_US", "zh-CN", "zh-TW"]
2024-11-07T22:24:06.656Z warning drconfig[01075] [SRM@6876 sub=vmomi.soapStub[413] opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007f5ca80b3510, h:15, <TCP '10.#.#.# : 53950'>, <TCP '10.#.#.# : 443'>>), /sdk>, method: registerExtension; code: 500(Internal Server Error)
2024-11-07T22:24:06.657Z error drconfig[01075] [SRM@6876 sub=LocalRegistrationManager opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] MethodFault error: Fault cause: vmodl.fault.SecurityError
-->
2024-11-07T22:24:06.657Z warning drconfig[01075] [SRM@6876 sub=LocalRegistrationManager opID=fca75d97-f3bb-4cfd-8a74-20a3b7c1ea40-configure:2c2b] Error while registering extension:
--> (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007f5ca80b3510, h:15, <TCP '10.#.#.# : 53950'>, <TCP '10.#.#.# : 443'>>), /sdk>]: registerExtension
--> Access to perform the operation was denied."
 
2024-11-07T22:24:06.592Z info vpxd[4153843] [Originator@6876 sub=MoExtensionMgr opID=26559664] Registering extension to Lookup service; com.vmware.vcDr, id: 7b03690d-ae19-48db-a565-9d0e6ca2c6d9_com.vmware.vcDr
2024-11-07T22:24:06.606Z info vpxd[4153843] [Originator@6876 sub=vmomi.soapStub[0] opID=26559664] SOAP request returned HTTP failure; <<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>, method: create; code: 500(Internal Server Error); fault: (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>]: create
--> "
--> }
2024-11-07T22:24:06.606Z warning vpxd[4153843] [Originator@6876 sub=LSClient opID=26559664] Service registration stub privilege error during lookup service RPC: N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
--> )
--> [context]zKq7AVECAQAAADAhbQEbdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsBoy8XbGlidm1vbWkuc28AAXKfJQElISABLk8gAZ3HHwF9NhoBzSoaAvbyAmxpYmxvb2t1cC10eXBlcy5zbwCDh/soAXZweGQAgwkAKQGDVeEPAYPjBWcCAdXDG4OKIkcCg6eWZQKD+aZlAoM
jvmQCg76QZQIA5ts3APk0OACT0FEEro4AbGlicHRocmVhZC5zby4wAAUv3g9saWJjLnNvLjYA[/context]
2024-11-07T22:24:06.607Z info vpxd[4153843] [Originator@6876 sub=LSClient opID=26559664] Refreshing lookup service token
2024-11-07T22:24:06.625Z info vpxd[4153843] [Originator@6876 sub=SsoClient opID=26559664] Successfully acquired token: SamlToken [subject={Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5; Domain:vsphere.local}, groups=[{Name:
Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=2024-11-07 22:24:06.607, expirationTime=2024-11-08 06:24:06.607, renewable=false, delegable=false, isSolution=true,confirmationType=1]
2024-11-07T22:24:06.638Z info vpxd[4153843] [Originator@6876 sub=vmomi.soapStub[0] opID=26559664] SOAP request returned HTTP failure; <<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>, method: create; code: 500(Internal Server Error); fault: (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>]: create
--> "
--> }
2024-11-07T22:24:06.638Z warning vpxd[4153843] [Originator@6876 sub=MoExtensionMgr opID=26559664] Failed to create LS service registration; id: 7b03690d-ae19-48db-a565-9d0e6ca2c6d9_com.vmware.vcDr, spec: (lookup.ServiceRegistration.CreateSpec) {


/var/log/vmware/vpxd/vpxd.log: 

2024-11-07T22:24:06.606Z info vpxd[4153843] [Originator@6876 sub=vmomi.soapStub[0] opID=26559664] SOAP request returned HTTP failure; <<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>, method: create; code: 500(Internal Server Error); fault: (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>]: create
--> "
--> }
2024-11-07T22:24:06.606Z warning vpxd[4153843] [Originator@6876 sub=LSClient opID=26559664] Service registration stub privilege error during lookup service RPC: N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
--> )
--> [context]zKq7AVECAQAAADAhbQEbdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsBoy8XbGlidm1vbWkuc28AAXKfJQElISABLk8gAZ3HHwF9NhoBzSoaAvbyAmxpYmxvb2t1cC10eXBlcy5zbwCDh/soAXZweGQAgwkAKQGDVeEPAYPjBWcCAdXDG4OKIkcCg6eWZQKD+aZlAoM
jvmQCg76QZQIA5ts3APk0OACT0FEEro4AbGlicHRocmVhZC5zby4wAAUv3g9saWJjLnNvLjYA[/context]
2024-11-07T22:24:06.607Z info vpxd[4153843] [Originator@6876 sub=LSClient opID=26559664] Refreshing lookup service token
2024-11-07T22:24:06.625Z info vpxd[4153843] [Originator@6876 sub=SsoClient opID=26559664] Successfully acquired token: SamlToken [subject={Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5; Domain:vsphere.local}, groups=[{Name:
Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=2024-11-07 22:24:06.607, expirationTime=2024-11-08 06:24:06.607, renewable=false, delegable=false, isSolution=true,confirmationType=1]
2024-11-07T22:24:06.638Z info vpxd[4153843] [Originator@6876 sub=vmomi.soapStub[0] opID=26559664] SOAP request returned HTTP failure; <<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>, method: create; code: 5
00(Internal Server Error); fault: (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>]: create
--> "
--> }
2024-11-07T22:24:06.638Z warning vpxd[4153843] [Originator@6876 sub=MoExtensionMgr opID=26559664] Failed to create LS service registration; id: 7b03690d-ae19-48db-a565-9d0e6ca2c6d9_com.vmware.vcDr, spec: (lookup.ServiceRegistration.CreateSpec) {


/var/log/vmware/lookupsvc/lookupserver-default.log: 

[2024-11-07T21:56:42.328Z pool-2-thread-115 INFO  com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-ada3f29a-XXXX-XXXX-XXXX-18e2393887c5, Domain: vsphere.local}

Environment

VMware Live Site Recovery 9.x
VMware Site Recovery Manager 8.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause


This is caused by an incorrect solution user in the vCenter Server configuration file located at: /etc/vmware-vpx/vpxd.cfg

1. Wrong domain associated with the VPXD solution user 
2. Machine ID mismatch 

If Site Recovery Manager finds the wrong Machine ID or SSO Domain in the vpxd.cfg file of the vCenter, extension registration will fail. 

How does an SSO domain cause this problem? 

Imagine having 2 independent vCenters (that is vCenters not in ELM)

1. OLD vCenter (SSO Domain: old.local)     = vpxd-1b90546f-####-####-####-########[email protected]
2. NEW vCenter (SSO Domain: new.local)  = vpxd-2b90446f-####-####-####-########[email protected]

When you decide to repoint the NEW vCenter to OLD vCenter SSO domain to create an Enhanced Linked Mode, the VPXD solution user of NEW vCenter doesn't get updated to @old.local and continues to exist as @new.local in the VPXD configuration file. This must be updated manually for you to be able to register any external solutions with vCenter. 

cat /etc/vmware-vpx/vpxd.cfg | less 

      <solutionUser>
        <certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
        <name>vpxd-2b90446f-####-####-####-########[email protected]</name>
        <privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
      </solutionUser>

vCenter Enhanced Linked Mode

Joining a vCenter Enhanced Linked Mode Domain

Understanding vSphere Domains and Domain Names

Repoint vCenter Server to Another vCenter Server in a Different Domain

Resolution


NOTE: Take powered OFF snapshots of vCenter before following the steps in this KB. Ignore the PSC, if you don't have one.  

VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice (85662)  

1.    Record the ESXi host on which vCenter and PSC is homed 
2.    Set DRS to manual mode for the clusters in which the hosts reside.
3.    Login to vCenter & PSC VAMI. Shutdown the vCenters first followed by the PSCs.
4.    Once all nodes are shutdown, snapshot VC & PSC from the host client. 
5.    Power ON the PSCs first followed by the vCenter 


1. Follow the Process to view the List of Services Registered with Single Sign-On

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk | less
/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk > /tmp/lookupsvc.txt


Use one of the commands above to extract the output below: 

    Name: AboutInfo.vpx.name
    Description: AboutInfo.vpx.name
    Service Product: com.vmware.cis
    Service Type: vcenterserver
    Service ID: a4d3d932-8381-4daa-9168-9a0ec5864685
    Site ID: default-first-site
    Node ID: 27f6891a-9255-4642-8889-4f0c74155ab6
    Owner ID: [email protected]
    Version: 8.0
    Endpoints:
        Type: com.vmware.vim.extension
        Protocol: vmomi
        URL: https://vcsa01.gslabs.local:443/sdkTunnel

The Service Type, Owner ID & URL belonging to the correct vCenter must be noted, if the environment is in ELM (Enhanced Linked Mode) with other vCenter servers. This will be the vCenter that you are trying to register SRM to but is failing. 

You can also use the dir-cli service list to list the solution users, but the above command must be run first to identify the correct solution user mapped to the vCenter you are looking for because this command lists all solutions users belonging to all the vCenters in ELM. 

 

1. machine-34952207-XXXX-XXXX-XXXX-3fb9f5c5a432
2. vsphere-webclient-34952207-c54e-4ea9-ada4-3fb9f5c5a432
3. vpxd-34952207-c54e-4ea9-ada4-3fb9f5c5a432
4. vpxd-extension-34952207-XXXX-XXXX-XXXX-3fb9f5c5a432
5. hvc-34952207-c54e-4ea9-ada4-3fb9f5c5a432
6. wcp-34952207-c54e-4ea9-ada4-3fb9f5c5a432
7. machine-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
8. vsphere-webclient-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
9. vpxd-65d0cec8-XXXX-XXXX-XXXX-cfaeab838226
10. vpxd-extension-65d0cec8-XXXX-XXXX-XXXX-cfaeab838226
11. hvc-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
12. wcp-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
13. com.vmware.vr-7c7b3860-4525-4f29-8e49-d80af6abe110

Make note of the Owner ID This will be required for updating the vpxd.cfg file in the following steps. 

Solution User format example

vpxd-34952207-XXXX-XXXX-XXXX-3fb9f5c5a432@vsphere.local

vpxd                              : Solution Username 
34952207-XXXX-XXXX-XXXX-3fb9f5c5a432: Machine ID 
vsphere.local                     : SSO Domain 


2. To clarify the Machine ID, you can run the following command locally on the vCenter Server node: 

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

The [email protected] Solution User ID must match the Machine ID machine-34952207-XXXX-XXXX-XXXX-3fb9fXXXX432 for that specific vCenter Server, if not there's a Machine ID mismatch. 

NOTE: When you list solution user certificates in large deployments, the output of /usr/lib/vmware-vmafd/bin/dir-cli list includes all solution users from all nodes. Run /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost to find the local machine ID for each host. Each solution username includes the machine ID. 

3. SSH into the vCenter, open the file using a text editor vi /etc/vmware-vpx/vpxd.cfg, and locate solutionUser

root@vcsa01 [ /etc/vmware-vpx ]#  vi vpxd.cfg

        <admin>[email protected]</admin>
        <isGroup>false</isGroup>
      </default>
      <groupcheck>
        <uri>https://vcsa01.gslabs.local/sso-adminserver/sdk/vsphere.local</uri>
      </groupcheck>
      <solutionUser>
        <certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
        <name>vpxd-34952207-XXXX-XXXX-XXXX-3fb9fXXXX432@vsphere.local</name>
        <privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
      </solutionUser>


4. The Solution User in vpxd.cfg must match the value of the Owner ID noted in Step 1. 

5. Backup the vpxd.cfg file - cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak

6. Modify the Machine ID or the SSO Domain depending on what you find wrong in this file. The Owner ID is found in Step 1. 

        <admin>[email protected]</admin>
        <isGroup>false</isGroup>
      </default>
      <groupcheck>
        <uri>https://vcsa01.gslabs.local/sso-adminserver/sdk/vsphere.local</uri>
      </groupcheck>
      <solutionUser>
        <certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
        <name>vpxd-34952207-XXXX-XXXX-XXXX-3fb9fXXXX432@vsphere.local</name>
        <privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
      </solutionUser>

7. Save changes and restart vCenter services - service-control --stop --all && service-control --start --all

NOTE: Do not restart vCenter services during backup activity (backup jobs will fail) or amidst other important vCenter activities like vMotion, sVMotion etc. 

Additional Information

Another way to check VPXD solution user from vCenter is to go to vCenter Configuration tab > Advanced settings > config.vpxd.sso.solutionUser.name

Impact/Risks:

The following resolution steps involve updating the vCenter Server solution user registered under /etc/vmware-vpx/vpxd.cfg. It is recommended that a backup of this file is performed at a minimum in the event you need to rollback changes.

Where vSphere Uses Certificates

VPXD.CFG file is not updating solution user with the new SSO domain name

Failed to register VRMS - Access to perform the operation was denied (312795)

Powered by Wolken Software