SRM fails to reconfigure - com.vmware.vapi.std.errors.unauthorized
Article ID: 312729
Updated On:
Products
Issue/Introduction
Symptoms:
SRM fails to reconfigure.
ERROR
Operation Failed
A general system error occurred: N7Vmacore9ExceptionE com.vmware.vapi.std.errors.unauthorized
Operation ID: 7c93d3af-ef79-4e2b-a377-756d6f325f26
1/4/24, 9:10:52 AM -0500
/var/log/vmware/dr/drconfig.log :
2024-01-04T14:10:49.780Z verbose drconfig[01237] [SRM@6876 sub=IO.Connection opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Attempting connection; <resolver p:0x00007f556400eae0,'adst-242-vc249.admin.mc.local:443', next:<TCP '142.222.242.249 : 443'>>, last e: 0(Success)
2024-01-04T14:10:49.963Z warning drconfig[00974] [SRM@6876 sub=CredentialsStore opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Credentials do not exist for key='SRM-1017bfc9-32e0-47dd-b54b-32deac6f0d66'2024-01-04T14:10:50.260Z error drconfig[00974] [SRM@6876 sub=ServiceAccountDomain opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] 'Create service account' error:
--> {
--> "ERROR": {
--> "com.vmware.vapi.std.errors.unauthorized": {
--> "data": {
--> "OPTIONAL": null
--> },
--> "error_type": {
--> "OPTIONAL": "UNAUTHORIZED"
--> },
--> "messages": [
--> {
--> "STRUCTURE": {
--> "com.vmware.vapi.std.localizable_message": {
--> "args": [],
--> "default_message": "Permission to perform this operation was denied.",
--> "id": "com.vmware.vapi.authorization.permission.denied",
--> "localized": {
--> "OPTIONAL": null
--> },
--> "params": {
--> "OPTIONAL": null
--> }
--> }
--> }
--> }
--> ]
--> }
--> }
--> }
2024-01-04T14:10:50.262Z verbose drconfig[00974] [SRM@6876 sub=VapiConnection opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] 'Delete session' completed. Result:--> null
2024-01-04T14:10:50.262Z verbose drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Logging out user '{Name: Administrator; Domain:VSPHERE.LOCAL}'
2024-01-04T14:10:50.262Z verbose drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Logging out session '527fa'
2024-01-04T14:10:50.265Z verbose drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Logged out session '527fa'.
2024-01-04T14:10:50.265Z info drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] User '{Name: Administrator; Domain:VSPHERE.LOCAL}' logged out.
2024-01-04T14:10:50.265Z verbose drconfig[00974] [SRM@6876 sub=vmomi.soapStub[61] opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Resetting stub adapter; <<last binding: <<TCP '142.222.242.248 : 46240'>, <TCP '142.222.242.249 : 443'>>>, /sdk>, (null)
2024-01-04T14:10:50.265Z verbose drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Logging out user '{Name: Administrator; Domain:VSPHERE.LOCAL}'
2024-01-04T14:10:50.277Z info drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] User '{Name: Administrator; Domain:VSPHERE.LOCAL}' logged out.
2024-01-04T14:10:50.277Z verbose drconfig[00974] [SRM@6876 sub=vmomi.soapStub[60] opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Resetting stub adapter; <<last binding: <<TCP '142.222.242.248 : 46238'>, <TCP '142.222.242.249 : 443'>>>, /sso-adminserver/sdk/vsphere.local>, (null)
2024-01-04T14:10:50.277Z verbose drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Logging out user '{Name: Administrator; Domain:VSPHERE.LOCAL}'
2024-01-04T14:10:50.277Z info drconfig[00974] [SRM@6876 sub=Default.UserSiteConnections opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] User '{Name: Administrator; Domain:VSPHERE.LOCAL}' logged out.
2024-01-04T14:10:50.277Z verbose drconfig[00974] [SRM@6876 sub=vmomi.soapStub[57] opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Resetting stub adapter; <<last binding: <<TCP '142.222.242.248 : 46176'>, <TCP '142.222.242.249 : 443'>>>, /lookupservice/sdk>, (null)
2024-01-04T14:10:50.277Z info drconfig[00974] [SRM@6876 sub=ClearOp opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Exiting ConfigureInfraNode
2024-01-04T14:10:50.277Z info drconfig[00974] [SRM@6876 sub=ClearOp opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Entering FixOwnership
2024-01-04T14:10:50.282Z info drconfig[00974] [SRM@6876 sub=ClearOp opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Exiting FixOwnership
2024-01-04T14:10:50.282Z error drconfig[00974] [SRM@6876 sub=ClearOp opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Operation failed
--> N7Vmacore9ExceptionE com.vmware.vapi.std.errors.unauthorized
--> [context]zKq7AVECAAQAAEWTVwEPZHJjb25maWcAAMrzG2xpYnZtYWNvcmUuc28AAch4BmxpYnZjLXV0aWwuc28AAmRuBGxpYnNybS1jZmcuc28AAmyIBAK0LAMDGgkMZHItY29uZmlndXJhdG9yAAMlzwwDAPgMA+EYDANvTAkA3kg1AOJhNQCwi0oEro4AbGlicHRocmVhZC5zby4wAAUv3g9saWJjLnNvLjYA[/context]
2024-01-04T14:10:50.282Z info drconfig[00974] [SRM@6876 sub=ClearOp opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] Exiting Start
2024-01-04T14:10:50.282Z verbose drconfig[01239] [SRM@6876 sub=DrConfigConfigurationManager ctxID=6d3bddf0 opID=83df1540-3351-47ff-9003-1d76a8ddfce9-configure:2028] OnError: Configuration task failed
--> (vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "N7Vmacore9ExceptionE com.vmware.vapi.std.errors.unauthorized"
--> msg = ""
--> }
/var/log/vmware/sso/websso.log :
2023-08-16T12:51:47.148Z INFO websso[83:tomcat-http--46] [CorId=96ca14a4-4d35-4519-8bf2-eade0f82d1de] [com.vmware.vcenter.tokenservice.clients.VapiClientConnection] Using cached stub for interface com.vmware.vcenter.identity.Providers
2023-08-16T12:51:47.170Z WARN websso[83:tomcat-http--46] [CorId=96ca14a4-4d35-4519-8bf2-eade0f82d1de] [com.vmware.vcenter.tokenservice.clients.VapiClientConnection] Caught exception invoking stub type interface com.vmware.vcenter.identity.Providers. Marking connection invalid so that it can be re-established. Exception was: Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.vapi.authorization.permission.denied,
defaultMessage = Permission to perform this operation was denied.,
args = [],
params = <null>,
localized = <null>
}],
data = <null>,
errorType = UNAUTHORIZED
}
2023-08-16T12:51:47.190Z WARN websso[83:tomcat-http--46] [CorId=96ca14a4-4d35-4519-8bf2-eade0f82d1de] [com.vmware.vcenter.tokenservice.clients.VapiClientConnection] Caught exception invoking stub type interface com.vmware.vcenter.identity.Providers. Marking connection invalid so that it can be re-established. Exception was: Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.vapi.authorization.permission.denied,
defaultMessage = Permission to perform this operation was denied.,
args = [],
params = <null>,
localized = <null>
}],
data = <null>,
errorType = UNAUTHORIZED
}
/var/log/vmware/sso/ssoAdminServer.log :
2024-01-04T14:10:51.614Z INFO ssoAdminServer[106:pool-2-thread-8] [OpId=728ffad0-03a4-442f-a32c-a4552252b0f0] [com.vmware.identity.vlsi.RoleBasedAuthorizer] User {Name: Administrator, Domain: VSPHERE.LOCAL} with role 'Administrator' is authorized for method call 'PrincipalManagementService.createLocalGroup'
2024-01-04T14:10:51.618Z INFO ssoAdminServer[103:pool-2-thread-5] [OpId=728ffad0-03a4-442f-a32c-a4552252b0f0] [auditlogger] {\"user\":\"[email protected]\",\"client\":\"\",\"timestamp\":\"01/04/2024 14:10:51 GMT\",\"description\":\"Creating local group 'SRM Administrators' with details ('null')\",\"eventSeverity\":\"INFO\",\"type\":\"com.vmware.sso.PrincipalManagement\"}
2024-01-04T14:10:51.618Z INFO ssoAdminServer[103:pool-2-thread-5] [OpId=728ffad0-03a4-442f-a32c-a4552252b0f0] [com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl] [User {Name: Administrator, Domain: VSPHERE.LOCAL} with role 'Administrator'] Creating local group 'SRM Administrators' with details ('null')
2024-01-04T14:10:51.630Z ERROR ssoAdminServer[103:pool-2-thread-5] [OpId=728ffad0-03a4-442f-a32c-a4552252b0f0] [com.vmware.identity.idm.server.IdentityManager] Failed to add group [SRM Administrators] in tenant [vsphere.local]
2024-01-04T14:10:51.630Z ERROR ssoAdminServer[103:pool-2-thread-5] [OpId=728ffad0-03a4-442f-a32c-a4552252b0f0] [com.vmware.identity.idm.server.ServerUtils] Exception 'com.vmware.identity.idm.InvalidPrincipalException: Another user or group SRM Administrators already exists with the same name'
com.vmware.identity.idm.InvalidPrincipalException: Another user or group SRM Administrators already exists with the same name
at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.addGroup(VMwareDirectoryProvider.java:3675) ~[vmware-identity-idm-server-7.0.0.jar:?]
at com.vmware.identity.idm.server.IdentityManager.addGroup(IdentityManager.java:5869) ~[vmware-identity-idm-server-7.0.0.jar:?]
at com.vmware.identity.idm.server.IdentityManager.addGroup(IdentityManager.java:10880) [vmware-identity-idm-server-7.0.0.jar:?]
at com.vmware.identity.idm.client.CasIdmClient.addGroup(CasIdmClient.java:2644) [vmware-identity-idm-client-7.0.0.jar:?]
at com.vmware.identity.admin.server.ims.impl.PrincipalManagementImpl.createLocalGroup(PrincipalManagementImpl.java:1540) [sso-adminserver-7.0.0.jar:?]
at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl$3.call(PrincipalManagementServiceImpl.java:143) [sso-adminserver-7.0.0.jar:?]
at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl$3.call(PrincipalManagementServiceImpl.java:125) [sso-adminserver-7.0.0.jar:?]
at com.vmware.identity.admin.vlsi.util.VmodlEnhancer.invokeVmodlMethod(VmodlEnhancer.java:186) [sso-adminserver-7.0.0.jar:?]
at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl.createLocalGroup(PrincipalManagementServiceImpl.java:125) [sso-adminserver-7.0.0.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_351]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_351]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_351]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_351]
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:99) [vlsi-server-7.0.0.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server-7.0.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_351]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_351]
2024-01-04T14:10:51.630Z ERROR ssoAdminServer[103:pool-2-thread-5] [OpId=728ffad0-03a4-442f-a32c-a4552252b0f0] [com.vmware.identity.admin.server.ims.impl.PrincipalManagementImpl] Error in createLocalGroup. Check if group already exists. Idm client exception.com.vmware.identity.idm.InvalidPrincipalException: Another user or group SRM Administrators already exists with the same name
2024-01-04T14:10:51.631Z INFO ssoAdminServer[103:pool-2-thread-5] [OpId=728ffad0-03a4-442f-a32c-a4552252b0f0] [com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl] The specified principal (SRM Administrators) is invalid.
com.vmware.vim.sso.admin.exception.InvalidPrincipalException: The specified principal (SRM Administrators) is invalid.
at com.vmware.identity.admin.server.ims.impl.PrincipalManagementImpl.createLocalGroup(PrincipalManagementImpl.java:1553) ~[sso-adminserver-7.0.0.jar:?]
at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl$3.call(PrincipalManagementServiceImpl.java:143) ~[sso-adminserver-7.0.0.jar:?]
at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl$3.call(PrincipalManagementServiceImpl.java:125) ~[sso-adminserver-7.0.0.jar:?]
at com.vmware.identity.admin.vlsi.util.VmodlEnhancer.invokeVmodlMethod(VmodlEnhancer.java:186) [sso-adminserver-7.0.0.jar:?]
at
com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl.createLocalGroup(PrincipalManagementServiceImpl.java:125) [sso-adminserver-7.0.0.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_351]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_351]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_351]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_351]
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:99) [vlsi-server-7.0.0.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server-7.0.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_351]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_351]
Caused by: com.vmware.identity.idm.InvalidPrincipalException: Another user or group SRM Administrators already exists with the same name
at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.addGroup(VMwareDirectoryProvider.java:3675) ~[vmware-identity-idm-server-7.0.0.jar:?]
at com.vmware.identity.idm.server.IdentityManager.addGroup(IdentityManager.java:5869) ~[vmware-identity-idm-server-7.0.0.jar:?]
at com.vmware.identity.idm.server.IdentityManager.addGroup(IdentityManager.java:10880) ~[vmware-identity-idm-server-7.0.0.jar:?]
at com.vmware.identity.idm.client.CasIdmClient.addGroup(CasIdmClient.java:2644) ~[vmware-identity-idm-client-7.0.0.jar:?]
at com.vmware.identity.admin.server.ims.impl.PrincipalManagementImpl.createLocalGroup(PrincipalManagementImpl.java:1540) ~[sso-adminserver-7.0.0.jar:?]
... 13 more
/var/log/vmware/srm/vmware-dr.log :
2024-01-08T15:34:32.610Z info drconfig[00974] [SRM@6876 sub=ServiceControl opID=36068fac-a38e-4867-b568-5f3939ab883a-configure:1c1a] Successfully retrieved status for service srm-vpostgres
2024-01-08T15:34:32.610Z info drconfig[00974] [SRM@6876 sub=SchemaUtils opID=36068fac-a38e-4867-b568-5f3939ab883a-configure:1c1a] DB Manager runtime version: 8.8.0
2024-01-08T15:34:32.611Z info drconfig[00974] [SRM@6876 sub=SchemaUtils opID=36068fac-a38e-4867-b568-5f3939ab883a-configure:1c1a] Current DB version: 8.6.0
/var/log/vmware/sso/svcaccountmgmt.log :
2024-01-04T14:10:52.384Z ERROR svcaccountmgmt[49:tomcat-http--12] [CorId=1a8aa47d-3e2e-4710-8c12-3073bad4b540 OpId=] [com.vmware.vcenter.svcaccountmgmt.vapi.setup.AuthzPermissionValidator] User VSPHERE.LOCAL\\Administrator who belongs to groups [vsphere.local\\CAAdmins, vsphere.local\\Everyone, vsphere.local\\SystemConfiguration.SupportUsers, vsphere.local\\Users, vsphere.local\\Administrators, vsphere.local\\SystemConfiguration.Administrators, vsphere.local\\ComponentManager.Administrators, vsphere.local\\SystemConfiguration.ReadOnly, vsphere.local\\LicenseService.Administrators, vsphere.local\\SystemConfiguration.BashShellAdministrators, vsphere.local\\HmsAdministrators] has no required privileges [ServiceAccount.ManageAccount, ServiceAccount.Administer] to invoke API com.vmware.vcenter.svcaccountmgmt.service_account.create
Environment
Cause
The ServiceAccount. * Permissions are not in the Administrator1 role; therefore, the user doesn't have them.
The user permissions override the permissions from groups, therefore the user 'VSPHERE.LOCAL\\Administrator' is not allowed to call the API.
Resolution
Follow the steps below depending on the errors you see in the logs & reconfigure SRM -
If the DB Manager runtime version doesn't match with the Current DB version, upgrade the DB by running the command.
/opt/vmware/srm/bin/initdb --cmd upgrade --cfg /opt/vmware/srm/conf/vmware-dr.xml
When you find the error - Another user or group SRM Administrators already exists with the same name -
1. Login vSphere Client -> Administratration -> User and Groups -> Groups -> Delete SRM Administrators and SRM Remote Users
2. Reconfigure SRM
In this scenario there was a global permission on the user [email protected] - it had the role Administrator1. In the past a problem was identified with the built-in administrator role during vCenter maintenance causing errors there was an additional administrator role called 'Administrator1' created through LDIF to resolve the errors in vCenter
Check the options mentioned in the screenshot -
Go to vCenter home > Administration > Service Account Management and check the options in the screenshot
Additional Information
How Do Multiple Permission Settings Work in vSphere
Impact/Risks:
When SRM reconfiguration fails after an upgrade, it cannot be used to perform DR activities because the sites cannot be reconnected.
Feedback
