Failed to register VRMS - Operation create is not permitted for VPXD solution user

search cancel

Failed to register VRMS - Operation create is not permitted for VPXD solution user

book

Article ID: 312795

calendar_today

Updated On:

Products

VMware Live Recovery VMware vCenter Server

Issue/Introduction

Symptoms:

Upon initial deployment of vSphere Replication, the plugin (com.vmware.vcHms) is not registered within the vCenter Server (Home > Administration > Client Plugins).

The following symptoms are present:

No stale extensions for vSphere Replication are present on the Platform Services Controller
- For more information on how to verify and remove stale extensions: Clean up the vCenter Server Extension Manager
Registering the vSphere Replication plugin against vCenter errors out with: "Bad exit code: 1"
No vSphere Replication (hms.log) log file is created after attempting to register
Running lsdoctor (KB80469) observed no Lookup Service errors
In /var/log/vmware/sso/lookupServer.log, the following message is present when attempting to register the plugin:

[2020-09-25T09:45:25.482Z pool-2-thread-4 vsphere.local 376753fb-92ee-4aae-90ba-130fa93ae275 INFO com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] Cannot find solution user [[email protected]] in [CN=ServicePrincipals,DC=vsphere,DC=local]
[...]
[2020-09-25T09:45:25.482Z pool-2-thread-4 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-37ee30a5-61be-452e-8aed-d2fbdb642cce, Domain: vsphere.local}

vCenter 6.x - /var/log/vmware/sso/lookupServer.log
vCenter 7.x - /var/log/vmware/lookupsvc/lookupServer.log

Trying to register vSphere Replication Appliance, fails with the error:

ERROR
Operation Failed
A general system error occurred: Failed to register VRMS.
Operation ID: 190e7416-c701-4a53-8b31-c52d7e3ed798
4/12/24, 12:03:48 PM -0600

/var/log/vmware/lookupsvc/lookupserver-default.log:

[2024-04-16T16:11:52.845Z pool-2-thread-5 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user <Anonymous>
[2024-04-16T16:11:52.845Z pool-2-thread-5 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user <Anonymous>
[2024-04-16T16:11:52.896Z pool-2-thread-5 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-2b90446f-e733-458a-1d32-a5a4a4f09ef0, Domain: old.local}

/var/log/vmware/vpxd/lookupserver-default.log:

2024-04-19T18:08:20.961Z warning vpxd[06358] [Originator@6876 sub=vmomi.soapStub[0] opID=1ce7966d] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007f9450c45458, h:7, <TCP '127.0.0.1 : 50746'>, <TCP '127.0.0.1 : 443'>>), /lookupservice/sdk>, method: create; code: 500(Internal Server Error)
2024-04-19T18:08:20.962Z warning vpxd[06358] [Originator@6876 sub=LSClient opID=1ce7966d] Service registration stub privilege error during lookup service RPC: N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
--> )
--> [context]zKq7AVECAQAAAG0mVQEbdnB4ZAAA9tg3bGlidm1hY29yZS5zbwAAjXgsAAtsLQAT6TIBnZcYbGlidm1vbWkuc28AAUtyDAF+GhUBbiQVAX66FAGpuhABqLUQAiSPAmxpYmxvb2t1cC10eXBlcy5zbwADhOmDdnB4ZAADKO6DA3QcgIRxUgUBbGlidmltLXR5cGVzLnNvAIPuyGEBg+nLYAGDKs1gAYNY3GABg7sJYAGDhrNgAQCnSSMANZ8jALRkNwWHfwBsaWJwdGhyZWFkLnNvLjAABi82D2xpYmMuc28uNgA=[/context]
2024-04-19T18:08:21.010Z warning vpxd[06358] [Originator@6876 sub=Vmomi opID=1ce7966d] VMOMI activation LRO failed; <<5263d1dd-c765-cf64-8a40-4d1167d12049, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 45920'>>,
ExtensionManager, vim.ExtensionManager.registerExtension>, N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
--> )
--> [context]zKq7AVECAQAAAG0mVQEbdnB4ZAAA9tg3bGlidm1hY29yZS5zbwAAjXgsAAtsLQAT6TIBnZcYbGlidm1vbWkuc28AAUtyDAF+GhUBbiQVAX66FAGpuhABqLUQAiSPAmxpYmxvb2t1cC10eXBlcy5zbwADOeyDdnB4ZAADKO6DA3QcgIRxUgUBbGlidmltLXR5cGVzL
nNvAIPuyGEBg+nLYAGDKs1gAYNY3GABg7sJYAGDhrNgAQCnSSMANZ8jALRkNwWHfwBsaWJwdGhyZWFkLnNvLjAABi82D2xpYmMuc28uNgA=[/context]
2024-04-19T18:08:21.012Z info vpxd[06358] [Originator@6876 sub=vpxLro opID=1ce7966d] [VpxLRO] -- FINISH lro-1484155
2024-04-19T18:08:21.021Z info vpxd[06320] [Originator@6876 sub=vpxLro opID=65f5f5ba] [VpxLRO] -- BEGIN lro-1484166 -- SessionManager -- vim.SessionManager.logout -- 5263d1dd-c765-cf64-8a40-4d1167d12049(525b4295
-a3b4-04bd-7e74-f7e8a72e377a)
2024-04-19T18:08:21.022Z info vpxd[06320] [Originator@6876 sub=vpxLro opID=65f5f5ba] [VpxLRO] -- FINISH lro-1484166
2024-04-19T18:08:21.026Z info vpxd[06358] [Originator@6876 sub=Default opID=1ce7966d] [VpxLRO] -- ERROR lro-1484155 -- ExtensionManager -- vim.ExtensionManager.registerExtension: vmodl.fault.SecurityError:
--> Result:
--> (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
2024-04-19T17:09:47.975Z warning vpxd[06345] [Originator@6876 sub=Vmomi opID=406cc114] VMOMI activation LRO failed; <<52457667-3880-5be8-f425-f203190ffdbf, <TCP '127.0.0.1 : 8085'>, <
TCP '127.0.0.1 : 45920'>>, ProxyService, vim.ProxyService.addEndpoint>, N3Vim5Fault13AlreadyExists9ExceptionE(Fault cause: vim.fault.AlreadyExists
--> )
--> [context]zKq7AVECAQAAAG0mVQEUdnB4ZAAA9tg3bGlidm1hY29yZS5zbwAAjXgsAAtsLQAT6TIBwaFvdnB4ZAABChV+AWAufgE8L36CkpcFAWxpYnZpbS10eXBlcy5zbwCB7shhAYHpy2ABgSrNYAGBWNxgAYG7CWABgYazYAEAp0kjAD
WfIwC0ZDcDh38AbGlicHRocmVhZC5zby4wAAQvNg9saWJjLnNvLjYA[/context]
2024-04-19T17:09:47.977Z info vpxd[06345] [Originator@6876 sub=vpxLro opID=406cc114] [VpxLRO] -- FINISH lro-1461161
2024-04-19T17:09:47.977Z info vpxd[06345] [Originator@6876 sub=Default opID=406cc114] [VpxLRO] -- ERROR lro-1461161 -- ProxyService -- vim.ProxyService.addEndpoint: vim.fault.AlreadyExists:
--> Result:
--> (vim.fault.AlreadyExists) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "vcenter.vmware.com:8089"
-->    msg = ""
--> }
--> Args:
-->
--> Arg endpoint:
--> (vim.ProxyService.LocalTunnelSpec) {
-->    serverNamespace = "vcenter.vmware.com:8089",
-->    accessMode = "httpOnly",
-->    port = 8089
--> }
2024-04-19T17:09:48.048Z info vpxd[06905] [Originator@6876 sub=vpxLro opID=7951b773] [VpxLRO] -- BEGIN lro-1461163 -- ExtensionManager -- vim.ExtensionManager.registerExtension -- 524
57667-3880-5be8-f425-f203190ffdbf(5246bea9-b4f1-449e-cb75-03c986d08d19)
2024-04-19T17:09:48.049Z error vpxd[06905] [Originator@6876 sub=MoExtensionMgr opID=7951b773] Extension with key com.vmware.vcHms not found

/var/log/vmware/hms/hms-configtool.log:

2024-04-10 12:05:31.045 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | vSphere Replication Appliance configuration error:vCenter Server extension configuration problem.
2024-04-10 12:05:31.045 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | Details: Unable to register extension in vCenter Server.
2024-04-10 12:05:31.045 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | null
com.vmware.hms.config.error.VrConfigException: Unable to register extension in vCenter Server.
        at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:287) ~[vr-config-8.7.0.jar:?]
        at com.vmware.hms.config.VrConfig.expressSetup(VrConfig.java:347) ~[vr-config-8.7.0.jar:?]
        at com.vmware.hms.config.cli.command.ExpressSetup.run(ExpressSetup.java:58) ~[vr-config-tool-8.7.0.jar:?]
        at com.vmware.hms.config.cli.command.CommandBase.run(CommandBase.java:305) ~[vr-config-tool-8.7.0.jar:?]
        at com.vmware.hms.config.cli.App.run(App.java:151) ~[vr-config-tool-8.7.0.jar:?]
        at com.vmware.hms.config.cli.App.main(App.java:211) ~[vr-config-tool-8.7.0.jar:?]
Caused by: com.vmware.jvsl.cfg.ConfigException: Internal error.
        at com.vmware.jvsl.cfg.RegisterExtensionCommand.execute(RegisterExtensionCommand.java:153) ~[jvsl-ext-reg-8.7.0.jar:?]
        at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:280) ~[vr-config-8.7.0.jar:?]
        ... 5 more
Caused by: com.vmware.vim.binding.vmodl.fault.SecurityError: Access to perform the operation was denied.
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
        at java.lang.Class.newInstance(Unknown Source) ~[?:?]
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core-8.7.0.jar:?]
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core-8.7.0.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core-8.7.0.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167) ~[vlsi-core-8.7.0.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105) ~[vlsi-core-8.7.0.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92) ~[vlsi-core-8.7.0.jar:?]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86) ~[vlsi-core-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) ~[vlsi-client-8.7.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60) ~[vlsi-client-8.7.0.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
2024-04-10 12:05:31.051 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:287)
2024-04-10 12:05:31.051 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.hms.config.VrConfig.expressSetup(VrConfig.java:347)
2024-04-10 12:05:31.051 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.hms.config.cli.command.ExpressSetup.run(ExpressSetup.java:58)
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.hms.config.cli.command.CommandBase.run(CommandBase.java:305)
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.hms.config.cli.App.run(App.java:151)
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.hms.config.cli.App.main(App.java:211)
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | Caused by: com.vmware.jvsl.cfg.ConfigException: Internal error.
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.jvsl.cfg.RegisterExtensionCommand.execute(RegisterExtensionCommand.java:153)
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:280)
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] |        ... 5 more
2024-04-10 12:05:31.052 ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | Caused by: (vmodl.fault.SecurityError) {
   faultCause = null,
   faultMessage = null
}

Environment

VMware vCenter Server 6.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

This is caused by an incorrect solution user in the vCenter Server configuration file located at: /etc/vmware-vpx/vpxd.cfg

vSphere Replication attempts to register the plugin with the Solution User defined in vpxd.cfg. If the Solution User value is inaccurate, the plugin will not register.

How does an SSO domain cause this problem ?

Imagine having 2 independent vCenters (that is vCenters not in ELM)

1. OLD vCenter (SSO Domain: old.local) = [email protected]
2. NEW vCenter (SSO Domain: new.local) = [email protected]

When you decide to repoint the NEW vCenter to OLD vCenter SSO domain to create an Enhanced Linked Mode, the VPXD solution user of NEW vCenter doesn't get updated to @old.local and continues to exist as @new.local in the VPXD configuration file. This must be updated manually for you to be able to register any external solutions with vCenter.

cat /etc/vmware-vpx/vpxd.cfg | less

<solutionUser>
<certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
<name>[email protected]</name>
<privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
</solutionUser>

vCenter Enhanced Linked Mode

Joining a vCenter Enhanced Linked Mode Domain

Understanding vSphere Domains and Domain Names

Repoint vCenter Server to Another vCenter Server in a Different Domain

Resolution

Follow KB 2043509 to view the listed services that are registered to vCenter. The output would look like below for each entry:

Name: cs.keyvalue.servicenameresource
Description: cs.keyvalue.servicedescriptionresource
Service Product: com.vmware.cis
Service Type: cs.keyvalue
Service ID: 24803f6a-0865-4267-ab0a-fc47a74f5cf1_kv
Site ID: default-site
Node ID: c0d1ea69-49d4-45aa-b6cd-6a54588d8d13
Owner ID: [email protected]
Version: 1.0
Endpoints:
        Type: com.vmware.cis.kv.client
        Protocol: http
        URL: https://vCenterServer_FQDN_or_PNID:443/invsvc
        SSL trust: MIIEoDCCpROaVQHMbwECFmFlubr/f3R7qau3UU.......................s7XJJkS1ZW/q4lfpqSzZDfH8kT2yxonbQm9aJr0IWNY/R/J8KZa1a0jslE/wLfdDkNl

Note: You may need to pipe to less (| less) to view/navigate the output

Identify the entry that notes Service Type: vcenterserver, and confirm the endpoint URL is for the correct vCenter Server FQDN/IP.

This is important if the environment is in Enhanced Linked Mode with other vCenter Servers.

Make note of the “Owner ID:” This will be required for updating vCenter in the following steps. The solution user would typically be [email protected]

Each solution user format is as follows:

If you are unsure of the Machine ID, you can run the following command locally on the vCenter Server node: /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

The SolutionUser ID will machine ID for that specific vCenter Server following the vmafd-cli command

On vCenter, open the file /etc/vmware-vpx/vpxd.cfg, and navigate to the entries that look like below:

<solutionUser>
        <certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
        <name>[email protected]</name>
        <privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
      </solutionUser>
The Solution User in vpxd.cfg should match the value noted on Step 3 (Owner ID:). If it does not, continue with the steps below.
Make a backup of /etc/vmware-vpx/vpxd.cfg

cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak

Modify the highlighted entry with the “Owner ID:” value collected in Step 3

<solutionUser>
        <certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
        <name>[email protected]</name>
        <privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
      </solutionUser>

Save changes and restart vCenter Server services : service-control --stop --all && service-control --start --all

NOTE: Please don't run this command during backup activity (backup jobs will fail) or amidst other important vCenter activities like svMotion, etc.

Additional Information

Another way to check VPXD solution user from vCenter is to go to vCenter Configuration tab > Advanced settings > config.vpxd.sso.solutionUser.name

Impact/Risks:

The following resolution steps involve updating the vCenter Server solution user registered under /etc/vmware-vpx/vpxd.cfg. It is recommended that a backup of this file is performed at a minimum in the event you need to rollback changes.

Where vSphere Uses Certificates

Feedback

thumb_up Yes

thumb_down No