VMware Live Site Recovery (VLSR) Storage Replication Adapter (SRA) Issues

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

Often times it can be confusing when deciding where to seek assistance when administrators encounter storage related issues with Live Site Recovery, especially with Array Pairing, Protection Group creation/management, and Recovery Plan operations and failovers.
This KB is created to assist knowing how to navigate this between partner verified and supported products.
A SRA is a program that an array vendor provides that enables Live Site Recovery to work with a specific kind of array.
Reference : Using Array-Based Replication with VMware Live Site Recovery
For more than one type of storage array in use, SRA for each array is to be deployed on both of VMware Live Site Recovery Servers:
Refer : Compatibility Guide for SRA --- Site Recovery Manager (SRM) for SRA
Live Site Recovery (VLSR) is an orchestrator to register VMs into Protection Groups and Recovery Plans. The storage vendors Storage Replication Adapter shares with VLSR a list of Array Pairs with device pairs.
Storage Array is responsible for

- - LUN snapshots are not unmounted at one site
  - Promote and demote of datastores between sites does not complete or fail
  - Array Based Replication is not copying data from one site to other (Replication is broken)
  - The Test Failover doesn't create or delete copies of replica LUNs on target site
  - Reprotect is grayed-out after failover which means workflow reverseReplication on storage array has not completed or Storage Replication Adapter did not pass command request to VLSR
    
    eg: Failed to reverse replication for failed over devices. SRA command 'reverseReplication' failed.

NOTE : Broadcom download portal no longer holds Storage Array vendor's Storage Replication Adapter (SRA) .tar.gz files. To source SRA, contact the relevant replication vendor directly.

Environment

VMware Live Site Recovery 8.x
VMware Live Stie Recovery 9.x

Resolution

Array Based Replication is specific to a storage vendor and separate from Site Recovery User Interface, although information about replication (inquired through the SRA) is availalbe in VLSR GUI. Live Site Recovery has two options through Storage Replication Adapter after selecting the Array Pairs, which are Discover Array Pairs or Discover Devices. Site Recovery User Interface does not manipulate any consistency groups or change device pairs states on the storage array directly.

In vSphere Client or vSphere Web Client, click Site Recovery > Open Site Recovery.
On the Site Recovery home page, under the site pair box, and click View Details.
On the Site Pair tab, click Configure > Array Based Replication > Array Pairs.
Select an array pair and click Array Manager Pair > Discover Array Pairs to rescan the arrays, or Discover Devices to recompute the storage replicated devices.

On selecting array pair, the Array Pairs tab provides detailed information about all storage devices in the array, including local device name, the device it is paired with, the direction of replication, the protection group to which the device belongs, whether the datastore is local or remote, and replica pair identification for each replicated device. All of this data is gathered from the SRA via a discoveryArrays request, for example.

If an issue is observed with SRA, in any of the areas described above, it will be helpful to open a ticket with the Storage Vendor and seek the assistance from a trained SRA engineer. You can also make a likewise case with an VLSR engineer at VMware to assist collaboratively.

Storage Replication Adapters are software modules for Live Site Recovery that are developed and supported by storage partners of Broadcom and are distributed here by Broadcom with the permission of the storage partners. Storage Replication Adapters have separate End User License Agreements (EULAs) that are included within these downloads. These sort of partnerships are no different than other aspects with vSphere, such as drivers, hardware, and so forth.

The VMware Compatibility Guide for Live Site Recovery will help to identify the Storage Replication Adapter that is certified to work with both storage platform as well as Live Site Recovery.
Any issues found with the vendor provided SRAs are handled by corresponding vendor. Third-Party Software for replication is certified and supported by the providers of the software.
Contact SAN vendors regarding plans to support, vendors certify software, create a list of certified software in the VMware Compatibility Guide.

For more details about the Storage Replication Adapters and VMware vCenter Live Site Recovery, see the Add Storage Replication Adapters to the VMware Live Site Recovery Appliance

Additional Information

Array Based Replication software on array side, has the main function of establishing and maintaining LUN replication pairs. This is independent of VLSR entirely and could be used solely on its own. VLSR comes in as an orchestrator to direct (through the SRA) ABR software functions on replica pairs to be able to achieve an orchestrated failover, reverse replication, failback, DR tests, and so forth.

Steps are from bottom to top in the diagram below

1. Array Based Replication (ABR) consists of having a production LUN (source) replicated/paired with a destination copy (replica) that is in a demoted state on destination side.
2. Storage Replication Adapter (SRA) is storage vendor specific software that is installed inside VLSR appliance.
3. In Site Recovery UI - Add Array Pairs and select VM's associated with Device Pairs
4. In Site Recovery UI - Scan device pairs by clicking on Discover Devices
Select an array pair and click Array Manager Pair > Discover Array Pairs to rescan arrays, or Discover Devices to recompute storage devices and find/update replica pairs and their states.

The frequency Live Site Recovery performs regular array scans can be reconfigured by changing the storage.minDsGroupComputationInterval option in Advanced Settings in Site Recovery UI. See Change Storage Settings.

Array Based Replication (ABR) uses command requests through Storage Replication Adapter installed on the Live Site Recovery (VLSR). The command requests notifies a set of array operations from storage array to VLSR which steps have completed.

Important: Note that the command requests VLSR provides for all Storage Replication Adapter vendors are the same commands sent to all storage adapters, owned by any supported vendor. The storage vendor is responsible for how their SRA reacts to these requests (or directives) based upon specifics to their replication software and array itself.

Workflow parameters for Array operations:

syncOnce – requests immediate replication
querySyncStatus – queries the status of a replication initiated by sync
OnceocheckTestFailoverStart*– validates environment and target devices for test failover readiness
testFailoverStart – creates writable temporary copies of the target devices
testFailoverStop – deletes the temporary copies created by testFailoverStart
checkFailover*– validates environment and target devices for failover readiness
prepareFailover – makes source devices read-only and optionally takes a snapshot of the source devices in anticipation of a failover
failover – promotes target devices by stopping replication for those devices and making them writable
prepareReverseReplication and reverseReplication - reverses array replication so that the original target array becomes the source array and vice versa
prepareRestoreReplication and restoreReplication - restores array replication after failover.

VLSR executes the following commands to discover replicated storage:

discoverArrays – discover storage arrays configured for replication
discoverDevices – discover devices on a given storage array replicated to the specified target array

Example:
An array pair {00A, 00B}. The discoverDevices command from VLSR asks through SRA for the list of device pairs it configured. This helps to find out what devices are relevant to the current environment and allow to then setup Protection Groups for replica VMFS/RDM pairs.

Site A	Site B
DiscoverDeviceParameters ArrayId 00A PeerArrayId 00B	DiscoverDeviceParameters ArrayId 00B PeerArrayId 00A

All Array Based Replication between the two sites is managed by storage array. VLSR does not replicate data.
Live Site Recovery relies on the Storage Replication Adapter (SRA) to communicate with the storage array for all completed steps.
Each product (VLSR and the storage array) have its own set of workflow parameters and do not manipulate or change the others configuration.

If the problem is more of a datastore problem than a site recovery problem open a new support request for storage and collect ESXi host support bundles. How to file a Support Request in Customer Connect https://knowledge.broadcom.com/external/article/142884/

If skyline is installed and configured to monitor the system(s) check the storage health

If the storage is VMware vSAN 8 check the health of the datastores

command line: esxcli vsan cluster get and esxcli vsan health any issues engage the VMware vSAN team.

If the storage is a vmfs datastore, identify the disks used in SRA replication
Live on the system check the /var/run/logs/ directory for any storage errors: Troubleshooting storage issues.