VMware Live Site Recovery (VLSR) Storage Replication Adapter (SRA) Issues

Products

VMware Live Recovery

Issue/Introduction

Often times it can be confusing when deciding where to seek assistance when administrators encounter storage related issues with Live Site Recovery, especially with Array Pairing, Protection Group creation/management, and Recovery Plan operations and failovers.

The goal of this KB article is to assist with knowing how to navigate this between partner verified and supported products.

An SRA is a program that an array vendor provides that enables Live Site Recovery to work with a specific kind of array. reference

If you use more than one type of storage array, you must install the SRA for each type of array on both of the Live Site Recovery Server [...]

Symptoms:

Live Site Recovery (VLSR) is an orchestrator to register VM's into Protection Groups and Recovery Plans. The storage vendors Storage Replication Adapter shares with VLSR a list of Array Pairs with device pairs.

The Storage Array is responsible for

LUN snapshots are not unmounted at one site
Promote and Demote of the datastores between the sites does not complete or fail
Array Based Replication is not copying data from one site to the other (Replication is broken)
The Test Failover doesn't create or delete copies of the replica LUNs on the target site
Reprotect is grayed-out after failover which means the workflow reverseReplication on the storage array has not completed or the Storage Replication Adapter did not pass the command request to VLSR

Resolution

Array Based Replication is specific to the storage vendor and separate from the Site Recovery User Interface, although we do get information about replication (inquired through the SRA) in the VLSR GUI. Live Site Recovery has two options through the Storage Replication Adapter after selecting the Array Pairs, which are Discover Array Pairs or Discover Devices. Site Recovery User Interface does not manipulate any consistency groups or change device pairs states on the storage array directly.

In the vSphere Client or the vSphere Web Client, click Site Recovery > Open Site Recovery.
On the Site Recovery home page, under the site pair box, and click View Details.
On the Site Pair tab, click Configure > Array Based Replication > Array Pairs.
Select an array pair and click Array Manager Pair > Discover Array Pairs to rescan the arrays, or Discover Devices to recompute the storage replicated devices.

When you select an array pair, the Array Pairs tab provides detailed information about all the storage devices in the array, including the local device name, the device it is paired with, the direction of replication, the protection group to which the device belongs, whether the datastore is local or remote, and replica pair identification for each replicated device. All of this data is gathered from the SRA via a discoveryArrays request, for example.

If you encounter an SRA issue, in any of the areas described above, it will be helpful to open a ticket with the Storage Vendor and seek the assistance from a trained SRA engineer. You can also make a likewise case with an VLSR engineer at VMware to assist collaboratively.

Storage Replication Adapters are software modules for Live Site Recovery that are developed and supported by storage partners of VMware and are distributed here by VMware with the permission of the storage partners. Storage Replication Adapters have separate End User License Agreements (EULAs) that are included within these downloads. These sort of partnerships are no different than other aspects with vSphere, such as drivers, hardware, and so forth.

The VMware Compatibility Guide for Live Site Recovery will help you to identify the Storage Replication Adapter that is certified to work with both your storage platform and Live Site Recovery.
Any issues found with the vendor provided SRAs are handled by the corresponding vendor. Third-Party Software for replication is certified and supported by the providers of the software.
Please contact your SAN vendors regarding their plans to support, vendors certify software, we will create a list of certified software in the VMware Compatibility Guide.

For more details about the Storage Replication Adapters and VMware vCenter Live Site Recovery, please see the Add Storage Replication Adapters to the VMware Live Site Recovery Appliance

Additional Information

Array Based Replication software on the array side, has the main function of establishing and maintaining LUN replication pairs. This is independent of VLSR entirely and could be used solely on its own. VLSR comes in as an orchestrator to direct (through the SRA) ABR software functions on replica pairs to be able to achieve an orchestrated failover, reverse replication, failback, DR tests, and so forth.

Steps are from bottom to top in the diagram below

1. Array Based Replication (ABR) consists of having a production LUN (source) replicated/paired with a destination copy (replica) that is in a demoted state on the destination side.
2. Storage Replication Adapter (SRA) is the storage vendor specific software that is installed inside the VLSR appliance.
3. In the Site Recovery UI - Add Array Pairs and select the VM's associated with the Device Pairs
4. In the Site Recovery UI - Scan the device pairs by clicking on Discover Devices
Select an array pair and click Array Manager Pair > Discover Array Pairs to rescan the arrays, or Discover Devices to recompute the storage devices and find/update replica pairs and their states.

You can reconfigure the frequency Live Site Recovery performs regular array scans by changing the storage.minDsGroupComputationInterval option in Advanced Settings in the Site Recovery UI. See Change Storage Settings.

Array Based Replication (ABR) uses command requests through the Storage Replication Adapter installed on the Live Site Recovery (VLSR). The command requests notifies a set of array operations from the storage array to VLSR which steps have completed.

Important: Please note that the command requests VLSR provides for all Storage Replication Adapter vendors are the same commands sent to all storage adapters, owned by any supported vendor. The storage vendor is responsible for how their SRA reacts to these requests (or directives) based upon specifics to their replication software and array itself.

Workflow parameters for Array operations:

syncOnce – requests immediate replication
querySyncStatus – queries the status of a replication initiated by sync
OnceocheckTestFailoverStart*– validates environment and target devices for test failover readiness
testFailoverStart – creates writable temporary copies of the target devices
testFailoverStop – deletes the temporary copies created by testFailoverStart
checkFailover*– validates environment and target devices for failover readiness
prepareFailover – makes source devices read-only and optionally takes a snapshot of the source devices in anticipation of a failover
failover – promotes target devices by stopping replication for those devices and making them writable
prepareReverseReplication and reverseReplication - reverses array replication so that the original target array becomes the source array and vice versa
prepareRestoreReplication and restoreReplication - restores array replication after failover.

VLSR executes the following commands to discover replicated storage:

discoverArrays – discover storage arrays configured for replication
discoverDevices – discover devices on a given storage array replicated to the specified target array

Example:
An array pair {00A, 00B}. The discoverDevices command from VLSR will ask through the SRA for the list of device pairs it configured. From there we can find out what devices are relevant to the current environment and allow the admin to then setup Protection Groups for replica VMFS/RDM pairs.

Site A	Site B
DiscoverDeviceParameters ArrayId 00A PeerArrayId 00B	DiscoverDeviceParameters ArrayId 00B PeerArrayId 00A

All Array Based Replication between the two sites is managed by the storage array. VLSR does not replicate data.
Live Site Recovery relies on the Storage Replication Adapter (SRA) to communicate with the storage array for all completed steps.
Each product (VLSR and the storage array) have their own set of workflow parameters and do not manipulate or change the others configuration.

If the problem is more of a datastore problem than a site recovery problem open a new support request for storage and collect ESXi host support bundles. How to file a Support Request in Customer Connect https://knowledge.broadcom.com/external/article/142884/

If skyline is installed and configured to monitor the system(s) check the storage health

If the storage is VMware vSAN 8 check the health of the datastores

command line: esxcli vsan cluster get and esxcli vsan health any issues engage the VMware vSAN team.

If the storage is a vmfs datastore, identify the disks used in SRA replication
Live on the system check the /var/run/logs/ directory for any storage errors: Troubleshooting storage issues.

Impact/Risks: