After 9.0.4 converge VR plugin inaccessible and UI error: Unable to retrieve pairs from extension server at Unable to connect to HBR Management Server at https://test-srm01.com:443/vrms. Reason: Unexpected status code : 503

Products

• VMware Live Recovery

Issue/Introduction

Symptoms : 

• The VLR 9.0.4 Appliance with standard package was choose during the Upgrade even though the there is only SRM appliance which was converged during 9.0.4 VLR Appliance upgrade. 
  
  
• After performing a 904 converge and subsequently changing the IP address of the vSphere Replication (VLR) appliances in both the Data Center (DC) and Disaster Recovery (DR) sites, the following issues occur:

Error Message : Unable to retrieve pairs from extension server at https://test-srm01.com:443/vrms.Unable to connect to HBR Management Server at https://test-srm01.com:443/vrms. Reason: Unexpected status code : 503 

• The following error message is also observed in above scenario : 

Unable to retrieve pairs from extension server at https://test-srm01.com:443/vrms.Unable to connect to HBR Management Server at https://test-srm01.com:443/vrms. Reason: Unable to download versions file from HBR Management Server at https://test-srm01.com:443/vrms. Http response: HTTP/1.1 503 Service Unavailable.


test-srm01.com  VMware Live Site Recovery cannot connect to VR Management Server at 'https://test-srm01.com:443/vrms'. The connection to the remote server is down.
testdr-srm01.com  VMware Live Site Recovery cannot connect to VR Management Server at 'https://testdr-srm01.com:443/vrms'. The connection to the remote server is down.

• vSphere Replication Appliance is in inaccessible state in vSphere Client VLR Plugin page. 

Environment

VMware Live Site Recovery 9.0.4

Cause

There is a known issue in the VMware Live Site Recovery 9.0.4 appliance where IP address change to VLR appliance is not updated automatically to /opt/vmware/etc/hbr/hbrsrv-nic.xml file.

Resolution

Currently, there is no permanent resolution for this issue. This behavior is under investigation by Broadcom VMware engineering. To restore functionality, please follow the steps in the Workaround section below.

Workaround : 

To clear stale or incorrect IP addresses from the vSphere Replication (VLR) appliance configuration, perform the following steps on both the Protected (DC) and Recovery (DR) VLR appliances.

Prerequisites:

• Admin access to the VLR appliances via SSH.

• A snapshot of the VLR appliances is recommended before making manual file changes.

Procedure:

• Access the DC and DR VLR Appliances : Connect to DC and DR VLR appliance via putty session using admin credentials
  
  
• Verify Current Configuration : Navigate to the /opt/vmware/etc/hbr/hbrsrv-nic.xml file and validate the IP address in DR and 

Example : 

The following is the output which indicates the old IP address of VLR appliance in hbrsrv-nic.xml.  

[email protected] [ /opt/vmware/etc/hbr ]# cat hbrsrv-nic.xml
<config><ipForHMS /><ipForFilter>10.20.x.1</ipForFilter><ipForNFC /></config>[email protected] [ /opt/vmware/etc/hbr ]#

[e-mail] [ /opt/vmware/etc/hbr ]# cat hbrsrv-nic.xml
<config><ipForHMS /><ipForFilter>10.30.x.1</ipForFilter><ipForNFC /></config>root@testdr-srm01 [ /opt/vmware/etc/hbr ]#


• Backup and Edit the hbrsrv-nic.xml Configuration file :  If the old IP address is still showing up in /opt/vmware/etc/hbr/hbrsrv-nic.xml file then backup hbrsrv-nic.xml file and then remove the old IP address in /opt/vmware/etc/hbr/hbrsrv-nic.xml file in both DC and DR VLR Appliance.

Create a backup: cp hbrsrv-nic.xml hbrsrv-nic.xml.bak

Update the file : vi /opt/vmware/etc/hbr/hbrsrv-nic.xml and then remove the stale IP address. 

• Restart hms and hbrsrv Services : Restart the hms and hbrsrv services on both DC and DR VLR Appliance.

# systemctl restart hms  
# systemctl restart hbrsrv  

• Validation : Then validate whether the hbrsrv-nic.xml file in DC and DR VLR appliance reflects the correct IP address for VLR Appliance. 

Example : 

The following is the output which indicates the correct updated IP address of the VLR appliance in hbrsrv-nic.xml.  

# cat /opt/vmware/etc/hbr/hbrsrv-nic.xml

[email protected] [ /opt/vmware/etc/hbr ]# cat hbrsrv-nic.xml
<config><ipForHMS /><ipForFilter>10.40.x.1</ipForFilter><ipForNFC /></config>[email protected] [ /opt/vmware/etc/hbr ]#

[e-mail] [ /opt/vmware/etc/hbr ]# cat hbrsrv-nic.xml
<config><ipForHMS /><ipForFilter>10.50.x.1</ipForFilter><ipForNFC /></config>root@testdr-srm01 [ /opt/vmware/etc/hbr ]#

• Final UI Check : Post this change the vSphere Replication appliance should be in healthy state on the vLR appliance UI page. 

Example :