Registering or Reconfiguring vSphere Replication Appliance (VRMS) fails with the errors.
ERROR
Operation Failed
Access to perform the operation was denied.
Operation ID: 1c92fc5d-####-4a37-####-fba99128a2a6
10/17/24, 10:03:09 AM -0500
ERROR
Operation error
A general system error occurred: Failed to register VRMS.
Operation ID: cf14ec57-####-4ce9-####-59238d846a2d
10/15/24, 12:31:19 -0500
/var/log/vmware/vpxd/vpxd.log:
YYYY-MM-DD HH:MM:SS info vpxd[4153843] [Originator@6876 sub=vmomi.soapStub[0] opID=26559664] SOAP request returned HTTP failure; <<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>, method: create; code: 500(Internal Server Error); fault: (vmodl.fault.SecurityError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = "Received SOAP response fault from [<<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>]: create
--> "
--> }
YYYY-MM-DD HH:MM:SS warning vpxd[4153843] [Originator@6876 sub=LSClient opID=26559664] Service registration stub privilege error during lookup service RPC: N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
--> )
--> [context]zKq7AVECAQAAADAhbQEbdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsBoy8XbGlidm1vbWkuc28AAXKfJQElISABLk8gAZ3HHwF9NhoBzSoaAvbyAmxpYmxvb2t1cC10eXBlcy5zbwCDh/soAXZweGQAgwkAKQGDVeEPAYPjBWcCAdXDG4OKIkcCg6eWZQKD+aZlAoM
jvmQCg76QZQIA5ts3APk0OACT0FEEro4AbGlicHRocmVhZC5zby4wAAUv3g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS info vpxd[4153843] [Originator@6876 sub=LSClient opID=26559664] Refreshing lookup service token
YYYY-MM-DD HH:MM:SS info vpxd[4153843] [Originator@6876 sub=SsoClient opID=26559664] Successfully acquired token: SamlToken [subject={Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5; Domain:vsphere.local}, groups=[{Name:
Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=YYYY-MM-DD HH:MM:SS, expirationTime=YYYY-MM-DD HH:MM:SS, renewable=false, delegable=false, isSolution=true,confirmationType=1]
YYYY-MM-DD HH:MM:SS info vpxd[4153843] [Originator@6876 sub=vmomi.soapStub[0] opID=26559664] SOAP request returned HTTP failure; <<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>, method: create; code: 5
00(Internal Server Error); fault: (vmodl.fault.SecurityError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = "Received SOAP response fault from [<<cs p:00007f26605ac8e0, TCP:localhost:1080>, /lookupservice/sdk>]: create
--> "
--> }
YYYY-MM-DD HH:MM:SS warning vpxd[4153843] [Originator@6876 sub=MoExtensionMgr opID=26559664] Failed to create LS service registration; id: 7b03690d-ae19-48db-a565-9d0e6ca2c6d9_com.vmware.vcDr, spec: (lookup.ServiceRegistration.CreateSpec) {
--> serviceVersion = "8.8.0",
--> vendorNameResourceKey = <unset>,
--> vendorNameDefault = <unset>,
--> vendorProductInfoResourceKey = <unset>,
--> vendorProductInfoDefault = <unset>,
--> serviceEndpoints = (lookup.ServiceRegistration.Endpoint) [
--> (lookup.ServiceRegistration.Endpoint) {
--> url = "https://vrms.vmware.com:443/catalog/com.vmware.vcDr_catalog.zip",
--> endpointType = (lookup.ServiceRegistration.EndpointType) {
--> protocol = "https",
--> type = "com.vmware.cis.common.resourcebundle"
--> },
--> sslTrust = <unset>,
--> endpointAttributes = (lookup.ServiceRegistration.Attribute) [
--> (lookup.ServiceRegistration.Attribute) {
--> key = "com.vmware.cis.common.resourcebundle.basename",
--> value = "cis.vcextension.com_vmware_vcDr.ResourceBundle"
--> }
--> ]
--> }
--> ],
--> serviceAttributes = <unset>,
--> serviceNameResourceKey = "com.vmware.vcDr.label",
--> serviceNameDefault = <unset>,
--> serviceDescriptionResourceKey = "com.vmware.vcDr.summary",
--> serviceDescriptionDefault = <unset>,
--> ownerId = "[email protected]",
--> serviceType = (lookup.ServiceRegistration.ServiceType) {
--> product = "com.vmware.cis",
--> type = "com.vmware.vcDr"
--> },
--> nodeId = <unset>
--> }, N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
--> )
--> [context]zKq7AVECAQAAADAhbQEbdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsBoy8XbGlidm1vbWkuc28AAXKfJQElISABLk8gAZ3HHwF9NhoBzSoaAvbyAmxpYmxvb2t1cC10eXBlcy5zbwCDOP4oAXZweGQAgwkAKQGDVeEPAYPjBWcCAdXDG4OKIkcCg6eWZQKD+aZlAoMjvmQCg76QZQIA5ts3APk0OACT0FEEro4AbGlicHRocmVhZC5zby4wAAUv3g9saWJjLnNvLjYA[/context]
/var/log/vmware/vpxd/lookupserver-default.log:
YYYY-MM-DD HH:MM:SS pool-2-thread-119 INFO com.vmware.vim.lookup.vlsi.ServiceRegistrationImpl] User: {Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5, Domain: vsphere.local} attempted to delete not existing service with ID: 7b03690d-ae19-48db-a565-9d0e6ca2c6d9_com.vmware.vcDr
YYYY-MM-DD HH:MM:SS pool-2-thread-119 INFO com.vmware.vim.lookup.vlsi.ServiceRegistrationImpl] User: {Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5, Domain: vsphere.local} attempted to update not existing service with ID: 7b03690d-ae19-48db-a565-9d0e6ca2c6d9_com.vmware.vcDr
YYYY-MM-DD HH:MM:SS pool-2-thread-116 INFO com.vmware.vim.lookup.vlsi.ServiceRegistrationImpl] User: <Anonymous> attempted to get not existing service with ID: 7b03690d-ae19-48db-a565-9d0e6ca2c6d9_com.vmware.vcDr
YYYY-MM-DD HH:MM:SS pool-2-thread-115 INFO com.vmware.identity.token.impl.SamlTokenImpl] SAML token for SubjectNameId [[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
YYYY-MM-DD HH:MM:SS pool-2-thread-115 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5, Domain: vsphere.local}
YYYY-MM-DD HH:MM:SS pool-2-thread-115 INFO com.vmware.identity.token.impl.SamlTokenImpl] SAML token for SubjectNameId [[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
YYYY-MM-DD HH:MM:SS pool-2-thread-115 INFO com.vmware.identity.token.impl.SamlTokenImpl] SAML token for SubjectNameId [[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
YYYY-MM-DD HH:MM:SS pool-2-thread-115 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5, Domain: vsphere.local}
YYYY-MM-DD HH:MM:SS pool-2-thread-115 INFO com.vmware.identity.token.impl.SamlTokenImpl] SAML token for SubjectNameId [[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
YYYY-MM-DD HH:MM:SS pool-2-thread-115 INFO com.vmware.vim.lookup.vlsi.VlsiSecurityChecker] Operation create is not permitted for user {Name: vpxd-ada3f29a-a1a2-42fb-a49b-18e2393887c5, Domain: vsphere.local}
/var/log/vmware/hms/hms-configtool.log:
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | vSphere Replication Appliance configuration error:vCenter Server extension configuration problem.
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | Details: Unable to register extension in vCenter Server.
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | null
com.vmware.hms.config.error.VrConfigException: Unable to register extension in vCenter Server.
at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:287) ~[vr-config-8.7.0.jar:?]
at com.vmware.hms.config.VrConfig.expressSetup(VrConfig.java:347) ~[vr-config-8.7.0.jar:?]
at com.vmware.hms.config.cli.command.ExpressSetup.run(ExpressSetup.java:58) ~[vr-config-tool-8.7.0.jar:?]
at com.vmware.hms.config.cli.command.CommandBase.run(CommandBase.java:305) ~[vr-config-tool-8.7.0.jar:?]
at com.vmware.hms.config.cli.App.run(App.java:151) ~[vr-config-tool-8.7.0.jar:?]
at com.vmware.hms.config.cli.App.main(App.java:211) ~[vr-config-tool-8.7.0.jar:?]
Caused by: com.vmware.jvsl.cfg.ConfigException: Internal error.
at com.vmware.jvsl.cfg.RegisterExtensionCommand.execute(RegisterExtensionCommand.java:153) ~[jvsl-ext-reg-8.7.0.jar:?]
at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:280) ~[vr-config-8.7.0.jar:?]
... 5 more
Caused by: com.vmware.vim.binding.vmodl.fault.SecurityError: Access to perform the operation was denied.
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at java.lang.Class.newInstance(Unknown Source) ~[?:?]
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core-8.7.0.jar:?]
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core-8.7.0.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core-8.7.0.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167) ~[vlsi-core-8.7.0.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105) ~[vlsi-core-8.7.0.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92) ~[vlsi-core-8.7.0.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86) ~[vlsi-core-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:284) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:241) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:286) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) ~[vlsi-client-8.7.0.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:60) ~[vlsi-client-8.7.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:287)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.hms.config.VrConfig.expressSetup(VrConfig.java:347)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.hms.config.cli.command.ExpressSetup.run(ExpressSetup.java:58)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.hms.config.cli.command.CommandBase.run(CommandBase.java:305)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.hms.config.cli.App.run(App.java:151)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.hms.config.cli.App.main(App.java:211)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | Caused by: com.vmware.jvsl.cfg.ConfigException: Internal error.
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.jvsl.cfg.RegisterExtensionCommand.execute(RegisterExtensionCommand.java:153)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | at com.vmware.hms.config.helper.VcHelper.registerExtension(VcHelper.java:280)
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | ... 5 more
YYYY-MM-DD HH:MM:SS ERROR com.vmware.hms.config.cli.App [main] (..jvsl.util.LoggingErrorStream) [] | Caused by: (vmodl.fault.SecurityError) {
faultCause = null,
faultMessage = null
}
This is caused by an incorrect solution user in the vCenter Server configuration file located at: /etc/vmware-vpx/vpxd.cfg
1. Wrong domain associated with the VPXD solution user
2. Machine ID mis-match
NOTE: Take powered OFF snapshots of vCenter before following the steps in this KB. Ignore the PSC, if you don't have one.
1. Record the ESXi host on which vCenter and PSC is homed
2. Set DRS to manual mode for the clusters in which the hosts reside.
3. Login to vCenter & PSC VAMI. Shutdown the vCenters first followed by the PSCs.
4. Once all nodes are shutdown, snapshot VC & PSC from the host client.
5. Power ON the PSCs first followed by the vCenter
1. Follow the Process to view the List of Services Registered with Single Sign-On
/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk | less
/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk > /tmp/lookupsvc.txt
Use one of the commands above to extract the output below:
Name: AboutInfo.vpx.name
Description: AboutInfo.vpx.name
Service Product: com.vmware.cis
Service Type: vcenterserver
Service ID: a4d3d932-8381-4daa-9168-9a0ec5864685
Site ID: default-first-site
Node ID: 27f6891a-9255-4642-8889-4f0c74155ab6
Owner ID: [email protected]
Version: 8.0
Endpoints:
Type: com.vmware.vim.extension
Protocol: vmomi
URL: https://vcsa01.gslabs.local:443/sdkTunnel
You can also use the dir-cli service list
to list the solution users but the above command must be run first to identify the correct solution user mapped to the vCenter you are looking for because this command lists all the solutions users belonging to all the vCenters in ELM.
root@VCSA [ /usr/lib/vmware-vmafd/bin ]# ./dir-cli service list
Enter password for [email protected]:
1. machine-34952207-c54e-4ea9-ada4-3fb9f5c5a432
2. vsphere-webclient-34952207-c54e-4ea9-ada4-3fb9f5c5a432
3. vpxd-34952207-c54e-4ea9-ada4-3fb9f5c5a432
4. vpxd-extension-34952207-c54e-4ea9-ada4-3fb9f5c5a432
5. hvc-34952207-c54e-4ea9-ada4-3fb9f5c5a432
6. wcp-34952207-c54e-4ea9-ada4-3fb9f5c5a432
7. machine-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
8. vsphere-webclient-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
9. vpxd-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
10. vpxd-extension-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
11. hvc-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
12. wcp-65d0cec8-8d9e-4f3e-ba8b-cfaeab838226
13. com.vmware.vr-7c7b3860-4525-4f29-8e49-d80af6abe110
Make note of the Owner ID
This will be required for updating the vpxd.cfg file in the following steps.
Solution User format example
vpxd-34952207-c54e-4ea9-ada4-3fb9f5c5a432@vsphere.local
vpxd : Solution User Name
34952207-c54e-4ea9-ada4-3fb9f5c5a432: Machine ID
vsphere.local :SSO Domain
2. To clarify the Machine ID, you can run the following command locally on the vCenter Server node:
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
The [email protected]
Solution User ID must match the Machine ID machine-34952207-c54e-4ea9-ada4-3fb9f5c5a432
for that specific vCenter Server, if not there's a Machine ID mis-match.
NOTE: When you list solution user certificates in large deployments, the output of /usr/lib/vmware-vmafd/bin/dir-cli list
includes all solution users from all nodes. Run /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
to find the local machine ID for each host. Each solution user name includes the machine ID.
3. SSH into the vCenter, open the file using a text editor vi /etc/vmware-vpx/vpxd.cfg
, and locate solutionUser
root@vcsa01 [ /etc/vmware-vpx ]# vi vpxd.cfg
<admin>[email protected]</admin>
<isGroup>false</isGroup>
</default>
<groupcheck>
<uri>https://vcsa01.gslabs.local/sso-adminserver/sdk/vsphere.local</uri>
</groupcheck>
<solutionUser>
<certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
<name>vpxd-34952207-c54e-4ea9-ada4-3fb9f5c5a432@vsphere.local</name>
<privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
</solutionUser>
4. The Solution User in vpxd.cfg must match the value of the Owner ID
noted in Step 1.
5. Backup the vpxd.cfg file - cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak
6. Modify the Machine ID or the SSO Domain depending on what you find wrong in this file. The Owner ID
is found in Step 1.
<admin>[email protected]</admin>
<isGroup>false</isGroup>
</default>
<groupcheck>
<uri>https://vcsa01.gslabs.local/sso-adminserver/sdk/vsphere.local</uri>
</groupcheck>
<solutionUser>
<certificate>/etc/vmware-vpx/ssl/vcsoluser.crt</certificate>
<name>vpxd-34952207-c54e-4ea9-ada4-3fb9f5c5a432@vsphere.local</name>
<privateKey>/etc/vmware-vpx/ssl/vcsoluser.key</privateKey>
</solutionUser>
7. Save changes and restart vCenter services - service-control --stop --all && service-control --start --all
NOTE: Do not restart vCenter services during backup activity (backup jobs will fail) or amidst other important vCenter activities like vMotion, sVMotion etc.