vSphere Replication 9.x VAMI Page Inaccessible (503 Error)

Products

VMware Live Recovery

Issue/Introduction

Users may be unable to access the vSphere Replication Appliance Management Interface (VAMI) due to an HTTP 503 error. This issue prevents logging in or interacting with the VAMI, effectively blocking configuration changes and health monitoring through the web interface.

From VRMS, /var/log/vmware/drconfigui/dr-config.log

2024-09-30 14:38:02,978 [srm-reactive-thread-252] WARN  com.vmware.dr.configservice.summary.GetConnectedToCloudData 65635f66-b092-408f-b80d-bf34f8b9fa21 getConnectedToCloudData - Failed to get active agent:

com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:61)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:267)
        at com.vmware.vim.vmomi.client.http.impl.HttpAsyncExchange$1$2.invokeWithinScope(HttpAsyncExchange.java:166)
        at com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan.runWithinSpanContext(NoopTracer.java:120)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:17)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.lambda$wrap$1(ThreadContext.java:55)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.execute(ThreadContext.java:209)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.execute(ThreadContext.java:185)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.setupContext(ThreadContext.java:76)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.setupContext(ThreadContext.java:105)
        at com.vmware.dr.ui.tools.utilities.ExecutorUtils.lambda$wrap$1(ExecutorUtils.java:36)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

Environment

vSphere Replication 9.x (and potentially other versions)

Cause

The vSphere Replication appliance lost its registration or had an incorrect registration with the vCenter Server. This often manifests as a service-level communication failure, leading to the VAMI becoming unresponsive with a 503 error. Since the VAMI itself was inaccessible, standard GUI-based re-registration methods were not possible.

Resolution

The vSphere Replication appliance needs to be re-registered with the vCenter Server via the command line.

Steps to Re-register the vSphere Replication Appliance:

Access the Appliance Command Line:
- If SSH is currently disabled on the appliance, open the virtual machine console via your vSphere UI.
- Log in to the console with the root user credentials.
- To enable SSH (if needed for remote access for future troubleshooting): systemctl start sshd
Verify vCenter Server Configuration in hms-configuration.xml:
- Ensure the correct IP address or FQDN of your vCenter Server Appliance (VCSA) is listed in the hms-configuration.xml file on the VR appliance.
- Connect to the appliance via console or SSH.
- Navigate to the configuration directory: cd /opt/vmware/hms/conf
- Check for the hms-ls-url and hms-localvc-address entries:
  less hms-configuration.xml | grep -i hms-ls-url less hms-configuration.xml | grep -i hms-localvc-address
- Expected output example:
  <hms-ls-url>https://vcAddress:443/lookupservice/sdk</hms-ls-url>
  <hms-localvc-address>vcAddress</hms-localvc-address>
- Verify vcAddress matches your actual vCenter IP or FQDN. If incorrect, this indicates a potential configuration issue that needs manual editing of the hms-configuration.xml file (which is outside the scope of this KB, but is described in KB380705).
Execute the Re-registration Command:
- Run the following Python script to reconfigure and re-register the vSphere Replication appliance. You will need to replace the placeholder parameters with values specific to your environment.
```
/usr/bin/python /opt/vmware/share/htdocs/service/hms/cgi/hms-dr.py --cmd saveembeddedconf --ls https://<vcAddress>:443/lookupservice/sdk --lsthumbprint <vCenterThumbprint> --ssoadmin [email protected] --vcip <vcAddress_or_FQDN> --vcport 80 --vcthumbprint <vCenterThumbprint> --servername <sitename> --hmshost <hmsIP> --hmsport 8043 --self-moid <moid> --adminmail root@<hmsIP> --secmail root@<hmsIP> --strict-certs 1
```
Parameter Explanation and How to Find Them:
- <vcAddress> or <vcAddress_or_FQDN>: The IP address or FQDN of your vCenter Server Appliance.
- <vCenterThumbprint>: The SHA256 thumbprint of the vCenter Server's SSL certificate.
  - To obtain this from any machine that can connect to vCenter:
    openssl s_client -connect <vcenterfqdn>:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -sha256 -noout -in /dev/stdin
- <sitename>: The logical name of your vSphere Replication site as configured. This is the value used for the --servername parameter.
  - To obtain this from the VR appliance console/SSH:
    cat /opt/vmware/hms/conf/hms-configuration.xml | grep -i sitename
  - Look for the value enclosed within the <siteName> tags.
- <hmsIP>: The IP address of the vSphere Replication appliance you are currently working on.
- <moid>: The Managed Object ID (MOID) of the vSphere Replication virtual machine within vCenter Server.
  - To obtain this from the VR appliance console/SSH:
    cat /opt/vmware/etc/vami/ovfenv.xml
  - You'll need to parse the XML output. Look for an XML tag or attribute that contains an ID that starts with vm-. It's often found within a <Property> tag, such as <Property ovf:key="vami.instance.id" ovf:value="vm-XXXX"/>, where vm-XXXX is your MOID. EX: vm-12345

After successfully running the command, wait a few minutes for the services to re-register and initialize. You should then be able to access the vSphere Replication VAMI page (https://:5480) without the 503 error.