Site Recovery Manager shows the error message as "Unable to retrieve pairs from extension server"

Products

VMware vCenter Server 7.0 VMware Site Recovery Manager 8.x VMware Live Recovery

Issue/Introduction

Symptoms:

DR UI displays the below error:

Unable to retrieve pairs from extension server

After rebooting the Site Recovery Manager Appliance (SRM) or VMware Live Site Recoveree Appliance, the following error message appears on the DR UI:

Unable to retrieve pairs from extension server at https://<SRM FQDN>:443/drserver/vcdr/vmomi/sdk.
Unable to connect to Site recovery Manager Server at https://<SRM FQDN>:443/drserver/vcdr/vmomi/sdk Reason: Unable to download
versions file from Site Recovery Manager Server at https://<SRM FQDN>:443/drserver/vcdr/vmomi/sdk. HTTP responce; HTTP/1.1 503 Service Unavailable

Environment

vCenter Serer 7.0.X

vCenters using Enhanced Linked Mode
Site Recovery Manager 8.x
VMware Live Site Recovery 9.x

Cause

Group membership information for users of the Site Recovery Manager solution was inadvertently lost in the vCenter Server's VMDIR.

To check the status of group membership information by command into vCenter

1. SSH login vCenter
2. /usr/lib/vmware-vmafd/bin/dir-cli group list --name Administrators

example.
/usr/lib/vmware-vmafd/bin/dir-cli group list --name Administrators
Enter password for [email protected]:　<- vCenter login password　　
...
CN=SRM-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX,cn=ServicePrincipals,dc=vsphere,dc=local

1. SRM solution user ID is displayed in normal case.
2. In the case where vCenter Enhanced Linked Mode is being used between the protected site and DR site, two SRM Solution User IDs are displayed.
3. If SRM Solution User ID is not listed, the correct group memberships have been lost for SRM solution user.

To check the status of group membership information from logs

1. /var/log/vmware/srm/vmware-dr.log:

YYYY-MM-DDT|HH:MM:SS.806+09:00 info vmware-dr[00940] [SRM@6876 sub=SsoClient]
Successfully acquired token: SamlToken [subject={Name: SRM-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX; Domain:vsphere.local},
groups=[{Name: Everyone; Domain:vsphere.local}], 　← only one
delegationChain=[], startTime=YYYY-MM-DD HH:MM:SS.780, expirationTime=YYYY-MM-DD HH:MM:SS.780, renewable=false, delegable=false, isSolution=true,confirmationType=1]ger Server at https://SRM FQDN:443/drserver/vcdr/vmomi/sdk. HTTP responce; HTTP/1.1 503 Service Unavailable

Correct and expected group membership information is like the following :

YYYY-MM-DDT|HH:MM:SS.167+09:00 info vmware-dr[03838] [SRM@6876 sub=SsoClient opID=XXXXXXX-XXXX-XXXX-XXX-XXXXXXX-tryFederatedSso]
Successfully acquired token: SamlToken [subject={Name: SRM-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX; Domain:vsphere.local},
groups=[{Name: Administrators; Domain:vsphere.local}, 　　
{Name: SolutionUsers; Domain:vsphere.local}, 　　
{Name: LicenseService.Administrators; Domain:vsphere.local}, 　　
{Name: SystemConfiguration.Administrators; Domain:vsphere.local},
{Name: Everyone; Domain:vsphere.local}], 　　
delegationChain=[], startTime=YYYY-MM-DDT|HH:MM:SS.136, expirationTime=YYYY-MM-DDT|HH:MM:SS.136, renewable=false, delegable=false, isSolution=true,confirmationType=1]

2. vCenter /var/log/vmware/vpxd/vpxd.log:

YYYY-MM-DDT|HH:MM:SS.880+09:00 info vpxd[17273] [Originator@6876 sub=User opID=66f6bdd0:3767-fb]
SSO Login > User: 'VSPHERE.LOCAL\SRM-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX',
Groups: '{Name: Everyone; Domain:vsphere.local} ', <-- only one
DelegationChain: 'vsphere.local\vpxd-extension-YYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYY '

Correct and expected group membership information is like the following:

YYYY-MM-DDT|HH:MM:SS.751+09:00 info vpxd[17708] [Originator@6876 sub=User opID=XXXX-XXXX-XXXX-XXXX-XXXXX-f3] SSO Login > User: 'VSPHERE.LOCAL\SRM-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX', Groups:'{Name: Administrators; Domain:vsphere.local} {Name: SolutionUsers; Domain:vsphere.local} {Name: LicenseService.Administrators;Domain:vsphere.local} {Name: SystemConfiguration.Administrators; Domain:vsphere.local} {Name: Everyone; Domain:vsphere.local}', DelegationChain:'vsphere.local\vpxd-extension-YYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYY'