Configuring SRM fails with error "N7Vmacore9ExceptionE com.vmware.vapi.std.errors.internal_server_error"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• Reconfiguring SRM fails with the error:
  
  N7Vmacore9ExceptionE com.vmware.vapi.std.errors.internal_server_error

Environment

VMware Site Recovery Manager 8.x

VMware Site Recovery Manager 9.x

Cause

The Solution Users will be missing from the required groups for SRM configuration, resulting in missing necessary permissions to complete the SRM configuration.

Cause Validation:

• /opt/vmware/support/logs/dr-client/drconfig.log file indicates that "solution user unable to create service account since it could not validate the permission information"
  
  
  YYYY-MM-DDTHH:MM.SSSZ error drconfig[01013] [SRM@6876 sub=ServiceAccountDomain opID=f7716536-5e1a-4852-9803-4c424cfb042b-configure:b66e] 'Create service account' error:
  --> {
  -->     "ERROR": {
  -->         "com.vmware.vapi.std.errors.internal_server_error": {
  -->             "data": {
  -->                 "OPTIONAL": null
  -->             },
  -->             "error_type": {
  -->                 "OPTIONAL": "INTERNAL_SERVER_ERROR"
  -->             },
  -->             "messages": [
  -->                 {
  -->                     "STRUCTURE": {
  -->                         "com.vmware.vapi.std.localizable_message": {
  -->                             "args": [
  -->                                 "com.vmware.vcenter.svcaccountmgmt.service_account.create"
  -->                             ],
  -->                             "default_message": "Could not validate permission information for operation com.vmware.vcenter.svcaccountmgmt.service_account.create invocation.",
  -->                             "id": "com.vmware.vapi.authorization.permission.error",
  -->                             "localized": {
  -->                                 "OPTIONAL": null
  -->                             },
  -->                             "params": {
  -->                                 "OPTIONAL": null
  -->                             }
  -->                         }
  -->                     }
  -->                 }
  -->             ]
  -->         }
  -->     }
  --> }
  
  
• In the vCenter Server /var/log/vmware/sso/svcaccountmgmt.log, following errors are seen:
  
  YYYY-MM-DDTHH:MM.SSSZ ERROR svcaccountmgmt[82:tomcat-http--36] [CorId=########-####-####-####-############ OpId=] [com.vmware.vapi.authz.impl.AuthorizationFilter] Could not validate permission information for operation com.vmware.vcenter.svcaccountmgmt.service_account.create invocation.
com.vmware.vim.binding.vmodl.fault.SecurityError: null
  
  
• In the vCenter Server /var/log/vmware/vpxd-svcs/vpxd-svcs.log, following errors are seen:
  YYYY-MM-DDTHH:MM.SSSZ [authz-service-6 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=d625802d-064a-4459-a3f4-23b94e905b52 IS] User VSPHERE.LOCAL\serviceaccountmgmt-########-####-####-####-############ does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions
YYYY-MM-DDTHH:MM.SSSZ [authz-service-6 [] WARN  com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl  opId=d625802d-064a-4459-a3f4-23b94e905b52 IS] User VSPHERE.LOCAL\serviceaccountmgmt-########-####-####-####-############ does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions

• The output of below mentioned command confirms that user is missing from multiple groups.
  python solution_users_fixer.py --check
Checking group memberships for sps-########-####-####-####-############
  sps-########-####-####-####-############ is MISSING from groups:  {'cn=actasusers,dc=vsphere,dc=local', 'cn=solutionusers,dc=vsphere,dc=local'}
Checking group memberships for vsphere-ui-########-####-####-####-############
  vsphere-ui-########-####-####-####-############ is MISSING from groups:  {'cn=solutionusers,dc=vsphere,dc=local'}
  cms-########-####-####-####-############ is MISSING from groups:  {'cn=actasusers,dc=vsphere,dc=local', 'cn=caadmins,cn=builtin,dc=vsphere,dc=local', 'cn=solutionusers,dc=vsphere,dc=local', 'cn=serviceproviderusers,dc=vsphere,dc=local'}
Checking group memberships for vsphere-webclient-########-####-####-####-############
Checking group memberships for machine-########-####-####-####-############
Checking group memberships for vpxd-svc-acct-########-####-####-####-############
  vpxd-svc-acct-########-####-####-####-############ is MISSING from groups:  {'cn=actasusers,dc=vsphere,dc=local', 'cn=solutionusers,dc=vsphere,dc=local', 'cn=users,cn=builtin,dc=vsphere,dc=local', 'cn=systemconfiguration.administrators,dc=vsphere,dc=local', 'cn=licenseservice.administrators,dc=vsphere,dc=local', 'cn=serviceproviderusers,dc=vsphere,dc=local', 'cn=componentmanager.administrators,dc=vsphere,dc=local'}
Checking group memberships for vmware-vsm-########-####-####-####-############
  vmware-vsm-########-####-####-####-############ is MISSING from groups:  {'cn=actasusers,dc=vsphere,dc=local', 'cn=readonlyusers,dc=vsphere,dc=local', 'cn=solutionusers,dc=vsphere,dc=local', 'cn=serviceproviderusers,dc=vsphere,dc=local'}

 

Resolution

Download the attached script 'solution_users_fixer.py' on this KB article, and upload to the /root folder on the vCenter.
Then, use the file-moving utility of your choice (WinSCP for example) to copy the entire file to /root directory of the vCenter Server.

Initiate a SSH connection to the vCenter FQDN/IP on Port 22 using the PuTTY terminal.

Step 1 - Run the check function
The --check function compares the current group memberships against the version specific mapping defined in the script, and then prints out any missing or extra groups found. This does not make any changes.

python solution_users_fixer.py --check

Step 2 - Run the fix function
The --fix function makes the same comparison that --check does, but also updates the users group memberships to match the pre-defined set. As this directly makes edits to the vmdir database, ensure a snapshot or backup is taken before using this.

python solution_users_fixer.py --fix

Reconfigure the SRM.

Additional Information

Fixing missing SSO Group Memberships for vSphere Solution Users with the solution_users_fixer script