vSphere Replication Configuration Fails with Error 503: Unable to Establish TCP Connection Due to Hostname Misconfiguration
search cancel

vSphere Replication Configuration Fails with Error 503: Unable to Establish TCP Connection Due to Hostname Misconfiguration

book

Article ID: 401264

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms

When configuring vSphere Replication, the following issues and errors are observed:

  • Not all host-based replication servers appear in the Replication Servers tab within the vSphere Replication UI.

  • Repeated task failures in the vCenter Server client with the error: Config vSphere Replication failed with the following error: cannot establish TCP connection, unexpected status code: 503

  • vSphere Replication Management Server (VRMS) appliance was freshly deployed and registered successfully with vCenter and the cluster.

  • Configuration tasks fail with HTTP Error 503 (Service Unavailable):   Unexpected status code: 503

Network Verification:

  • Ports 443 and 31031 confirmed open between VRMS and ESXi hosts.

  • Ping and connectivity validated after network team verification.

Environment

VMware vSphere Replication 9.x

Cause

This issue occurs when hostname configuration for the vCenter Server and ESXi hosts is incorrect. Specifically:

  • vCenter Server and ESXi hosts were configured with the hostname localhost, instead of valid Fully Qualified Domain Names (FQDNs).
  • As a result, the VRMS appliance fails to resolve and establish communication with the required services, leading to the 503 errors during configuration.
Log Snippets
  • From the /var/log/vmware/vsphere-ui/vsphere_client_virgo.log on vCenter:

[2025-06-11T09:40:13.396+07:00] [ERROR] sdk-plugin-deployer-14023     com.vmware.vise.plugin.download.PluginDownloadServiceImpl         Downloading plug-in package 'com.vmware.drui.plugin:9.0.2.24401761' to
 '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vmware.drui.plugin-9.0.2.24401761/plugin-package.tmp' has failed due to unknown reasons. Removing failed plugin package folder '/etc/vmware/v
sphere-ui/vc-packages/vsphere-client-serenity/com.vmware.drui.plugin-9.0.2.24401761'. java.net.UnknownHostException: replication.appliance: Temporary failure in name resolution

  • From /opt/vmware/hms/logs/hms.log on the VRMS appliance:
    • The system attempted to enable replication traffic on this host using a NIC (vmk2) with only link-local IPs, but could not establish communication.

2025-06-11 12:02:33.433 ERROR com.vmware.hms.monitor.hostEnableHostAtHbrTaskRunner [hms-config-host-at-hbr-thread-1] (..monitor.host.EnableHostOnHbrHelper) [operationID=5f96e0b7-####-####-####-6dc9c473e617-HMSINT-8] | Failed to enable host10.##.##.150(host-##) for addresses [10.##.##.##, fe80::####:####:####:8cbc], using NICs [vSphereReplicationNFC.key-vim.host.VirtualNic-vmk2].
2025-06-11 12:02:33.433 ERROR com.vmware.hms.monitor.hostEnableHostAtHbrTaskRunner [hms-config-host-at-hbr-thread-1] (..jvsl.util.Slf4jUtil) [operationID=5f96e0b7-####-####-####-6dc9c473e617-HMSINT-8] | Failed to enable host 10.##.##.150(host-##) on any NIC in VR server replication.appliance(5216b772-####-####-####-3bb09834ad2b).
2025-06-11 12:02:33.433 ERROR com.vmware.hms.monitor.hostEnableHostAtHbrTaskRunner [hms-config-host-at-hbr-thread-1] (..host.task.EnableHostAtHbrTaskRunner) [operationID=5f96e0b7-####-####-####-6dc9c473e617-HMSINT-8] | Error while enabling host host-## in VR Server 127.#.#.#

    • HMS is trying to register the vSphere Replication server module on host, however the registration is getting failed

2025-06-11 12:02:38.599 ERROR com.vmware.hms.remote.SiteManager [hms-main-thread-8] (..hms.remote.SiteManagerImpl) [operationID=5f96e0b7-####-####-####-6dc9c473e617-HMSINT-8, task=HTID-de05272c-####-####-####-95b57b70c8fa] | Unable to register VR Server vm '10.##.##.150' uri 'https://10.##.##.150:443/hbr' thumbprint '56:D3:C9:CE:A7:25:3F:C9:63:##:##:##:##:44:AC:C4:99:FF:BF:9B:12:2A:A4:63:ED:94:EA:80'

    • The service hbrsrvuw on the host is not registering with HMS, which is needed for replication to work.

2025-06-11 12:02:38.602 ERROR com.vmware.hms.hbrsrvuw.HbrsrvuwRegistrarService [hms-main-thread-2] (..hms.hbrsrvuw.HbrsrvuwRegistrarService$HostTask) [operationID=5f96e0b7-####-####-####-6dc9c473e617-HMSINT-8] | Register hbrsrvuw from host-##...FAILED (HTID-de05272c-####-####-####-95b57b70c8fa); will retry

    • HMS is periodically pinging the HBR service on the host to check health and connectivity, however the ping is getting failed

2025-06-11 12:02:38.607 ERROR com.vmware.hms.net.hbr.ping.svr.7e38acc3-####-####-####-56f2526f4f25 [hms-ping-scheduled-thread-3] (..net.impl.VmomiPingConnectionHandler) [operationID=5f96e0b7-####-####-####-6dc9c473e617-HMSINT-8, operationID=3c1d4e2c-####-####-####-97f914e5078f-HMS-PING] | Ping for server 10.##.##.150:443/hbr for session: N/A failed:

Resolution

Workaround

  • Update Hostnames to Use FQDNs

    • Rename the vCenter Server to use a valid FQDN.
    • Rename all ESXi hosts to use valid FQDNs (avoid using localhost or short names).

  • Renew Certificates

    • After renaming, renew the SSL certificates for both vCenter Server and all ESXi hosts.

  • Reconfigure vSphere Replication

    • Reconfigure the vSphere Replication appliance.
    • Use the PSC FQDN during configuration.
    • Ensure the vSphere Replication appliance’s FQDN is resolvable in DNS.
Powered by Wolken Software