Download of VIB vmware-hbr-agent on ESXi host fails with error "Proxy server cannot connect to the endpoint: HTTP/1.1 503 Service Unavailable

Products

• VMware vCenter Server 8.0

Issue/Introduction

Multiple "Install" tasks for installation vmware-hbr-agent are getting pushed on ESXi host from vSphere Replication server.

Environment

• vCenter server running on version 8.0 U2
• vSphere Replication 
• Site recovery manager. 

Cause

• Review the host esxupdate.log, observe that the hbr vib download is getting failed with an error "Proxy server cannot connect to the endpoint: HTTP/1.1 503 Service Unavailable"
• vCenter was unable to resolve the shortname of the vSphere replication server.
• ESXi host is able to resolve the shortname of the vSphere replication server.
• Moreover, FQDN of the vSphere Replication server work properly from host and vCenter.

from esxupdate.log

YYYY-MM-DDThh:mm:ss.000Z In(14) esxupdate[133144951]: Opening https://vSphere_replication_server_hostname:8043/vib/vmware-hbr-agent.vib for download
YYYY-MM-DDThh:mm:ss.000Z In(14) esxupdate[133144951]: Proxies configured: {'http': 'http://vCenter.domain.com:80', 'https': 'http://vCenter.domain.com:80', 'ftp': 'http://vCenter.domain.com:80'}
YYYY-MM-DDThh:mm:ss.000Z Er(11) esxupdate[133144951]: Proxy server cannot connect to the endpoint: HTTP/1.1 503 Service Unavailable
YYYY-MM-DDThh:mm:ss.000Z Er(11)[+] esxupdate[133144951]: content-length: 19
YYYY-MM-DDThh:mm:ss.000Z Er(11)[+] esxupdate[133144951]: content-type: text/plain
YYYY-MM-DDThh:mm:ss.000Z Er(11)[+] esxupdate[133144951]: date: Thu, DD MM YYYY hh:mm:ss GMT
YYYY-MM-DDThh:mm:ss.000Z Er(11)[+] esxupdate[133144951]: connection:
YYYY-MM-DDThh:mm:ss.000Z Wa(12) esxupdate[133144951]: Download failed: Proxy server vCenter.domain.com:80 returned status 503, 1 retry left...
YYYY-MM-DDThh:mm:ss.000Z In(14) esxupdate[133144951]: Proxies configured: {'http': 'http://vCenter.domain.com:80', 'https': 'http://vCenter.domain.com:80', 'ftp': 'http://vCenter.domain.com:80'}

Perform the below test for verification, 

• Perform nslookup of the vSphere replication server's hostname (shortname) from vCenter server, it will fail.
• Perform nslookup of the vSphere replication server's with full FQDN, it works.

Resolution

1. Take SSH of the vCenter
2. Login with root credentials. 
3. Create an entry in the /etc/hosts file with shortname and FQDN of the vSphere replication server.
   • vi /etc/hosts
   • Add an entry for vSphere replication server IP address, FQDN and shortname.
   • :wq!      

OR 

1. Take SSH of the vCenter 
2. Login with root credentials.
3. Connect to VAMI config setting.
   • /opt/vmware/share/vami/vami_config_net
4. Select option 4 (DNS)
5. Change the domain name
   • add the domain name 
   • for eg: domain.com
6. Exit.