vSphere Replication fails to fetch the datastore information while configuring replication for VMs

Products

• VMware Live Recovery

Issue/Introduction

• Fails to fetch the datastore information in single direction while configuring replication for VMs. (One direction may work)
• Configure replication wizard displays an error as shown below:
  
  "ERROR
  Cannot get datastores information.
  Operation ID: ########-####-####-####-############"



• Configuring replication works for some time post reconfiguring the vSphere Replication.
  
  

Environment

VMware Site Recovery Manager (All versions)

vSphere Replication (All versions)

Cause

• The solution user "####.##\###.######.##-##-############-####-####-############" have only single administrator-level permission, whereas two permissions are required to retrieve datastore information during replication setup from both directions.
• This lack of privileges led to a failure in datastore discovery when configuring replication in one direction.

Cause validation:

• Navigate to vCenter > Permissions and observe the user permissions.
  
  Above screenshot confirms that the user having only single administrator-level permission.
  
  
• /opt/vmware/dr-client/dr.log file confirmed that the VC property collector failed to retrieve datastore details.
  YYYY-MM-DD TT:MM:SS,ssz [http-nio-###.#.#.#-8443-exec-1] INFO  com.vmware.srm.client.infrastructure.requestHandlers.impl.DispatcherServlet ################### ########-####-####-####-############ getDatastoresList - Processing request to 'getDatastoresList'
YYYY-MM-DD TT:MM:SS,ssz [srm-reactive-thread-3970] WARN  com.vmware.srm.client.infrastructure.pc.utils.PCUtil ################### ########-####-####-####-############ getDatastoresList - Retrieve properties failed for server ########-####-####-####-############, specs [[(vmodl.query.FilterSpec) {
YYYY-MM-DD TT:MM:SS,ssz [srm-reactive-thread-3978] ERROR com.vmware.dr.client.shared.datastores.DatastoresListDataHandler ################### ########-####-####-####-############ getDatastoresList - Cannot retrieve datastores for node: group-d1 at server: ########-####-####-####-############.com.vmware.vim.vmomi.client.exception.VmomiProtocolException: https://########.####.########.##.##:443/sdk invocation failed with "com.vmware.vim.vmomi.core.exception.UnmarshallException: Missing value for non-optional field vm"
YYYY-MM-DD TT:MM:SS,ssz [srm-reactive-thread-3978] WARN  com.vmware.dr.ui.tools.reactive.impl.PromiseImpl ################### ########-####-####-####-############ getDatastoresList - Function 'com.vmware.dr.client.shared.datastores.DatastoresListDataHandler$$Lambda/0x################@########' failed.java.lang.RuntimeException: Cannot get datastores information.
YYYY-MM-DD TT:MM:SS,ssz [srm-reactive-thread-3978] WARN  com.vmware.dr.client.shared.datastores.DatastoresListDataHandler ################### ########-####-####-####-############ getDatastoresList - DrRequestHandlerError:java.lang.RuntimeException: Cannot get datastores information.


• /var/log/vmware/vpxd-svcs/vpxd-svc.log file confirmed that the user "#########\###.######.##-##-########-####-####-####-############" does not have privileges.
  YYYY-MM-DDTTT:MM:SS.ssz [dataservice-1 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=da243b56-333d-40f0-9320-############ IS] User #########\###.######.##-##-########-####-####-####-############ does not have privileges [System.Read] on object urn%3Avmomi%3AInventoryServiceTag%##########-####-####-####-############%########
YYYY-MM-DDTTT:MM:SS.ssz [dataservice-2 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=e8bb33dd-7d0a-481f-9714-############ IS] User #########\###.######.##-##-########-####-####-####-############ does not have privileges [System.Read] on object urn%3Avmomi%3AInventoryServiceTag%##########-####-####-####-############%########
YYYY-MM-DDTTT:MM:SS.ssz [dataservice-1 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=e727da02-9e9f-46b3-890a-############ IS] User #########\###.######.##-##-########-####-####-####-############ does not have privileges [System.Read] on object urn%3Avmomi%3AInventoryServiceTag%##########-####-####-####-############%########

Resolution

Resolution 1:
-------------------
1. Restart vmware-vpxd on the vCenter(s)
"service-control --restart vmware-vpxd" and then try to configure replication.


Resolution 2:
-------------------
If the vCenters are part of the same SSO domain, both Solution Users and both Service Principals need to have Administrator permissions on both vCenters (propogated).

• vCenter object -> Permissions -> vsphere.local
• Check for the users mentioned in vpxd-svcs.log. There may be two similar users ending with the same ID, add the permissions as needed.

 

If the above does not resolve the issue, attempt to re-create the solution users and service principals by re-registering VRMS.

• Unregister the extension 'com.vmware.vcHms' from vcenter MOB page.
  Refer to: Clean up the vCenter Server Extension Manager
• Remove the existing user "###.######.##-##-########-####-####-####-############" from vcenter.
• Log into the lookup service using JXplorer and deleted the corresponding solution user entries under Service Principals > <ID from vpxd-svcs.log>
  Refer to: Removal of stale entries from VC using JXplorer
• Reconfigure both vSphere Replication appliances.
• Newly created user will now be granted with both necessary permissions.
  
• Configure Replication wizard to successfully retrieve the datastore list while configuring replication.