Hosts fail to install vmware-hbr-agent after vSphere Replication deployment
Article ID: 382388
Updated On:
Products
VMware Live Recovery
Issue/Introduction
Symptoms:
- Following error observed in vSphere Client (every minute):
"An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 7" - Below stated Task observed in host:
Task Name - Install
Status - An error occurred during host configuration.
Initiator - com.vmware.vr-sa-########-####-####-#####-#######@vsphere.local
- Noticed following errors in /var/log/esxupdate.log:
2024-11-13T07:00:24Z esxupdate: 5076018: esxupdate: INFO: nosigcheck = None2024-11-13T07:00:25Z esxupdate: 5076018: HostImage: INFO: Installers initiated are {'live': <vmware.esximage.Installer.LiveImageInstaller.LiveImageInstaller object at 0x8ad68f1610>, 'boot': <vmware.esximage.Installer.BootBankInstaller.BootBankInstaller object at 0x8ad6b092b0>, 'locker': <vmware.esximage.Installer.LockerInstaller.LockerInstaller object at 0x8ad68f18e0>}2024-11-13T07:00:25Z esxupdate: 5076018: imageprofile: INFO: Adding VIB VMware_locker_tools-light_12.2.6.22229486-22348808 to ImageProfile (Updated) ################2024-11-13T07:00:25Z esxupdate: 5076018: imageprofile: DEBUG: Adding Component VMware-VM-Tools_12.2.6.22229486-22348808 to ImageProfile (Updated) ################2024-11-13T07:00:25Z esxupdate: 5076018: downloader: INFO: Opening https://VR :443/vib/vmware-hbr-agent.vib for download2024-11-13T07:00:58Z esxupdate: 5076018: downloader: WARNING: Download failed: <urlopen error _ssl.c:1114: The handshake operation timed out>, 4 retry left...2024-11-13T07:01:31Z esxupdate: 5076018: downloader: WARNING: Download failed: <urlopen error _ssl.c:1114: The handshake operation timed out>, 3 retry left...2024-11-13T07:02:04Z esxupdate: 5076018: downloader: WARNING: Download failed: <urlopen error _ssl.c:1114: The handshake operation timed out>, 2 retry left...2024-11-13T07:02:37Z esxupdate: 5076018: downloader: WARNING: Download failed: <urlopen error _ssl.c:1114: The handshake operation timed out>, 1 retry left...2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: An esxupdate error exception was caught:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: Traceback (most recent call last):2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/urllib/request.py", line 1354, in do_open2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/http/client.py", line 1259, in request2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/http/client.py", line 1305, in _send_request2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/http/client.py", line 1254, in endheaders2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/http/client.py", line 1014, in _send_output2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/http/client.py", line 954, in send2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/http/client.py", line 1428, in connect2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/ssl.py", line 500, in wrap_socket2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/ssl.py", line 1040, in _create2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/ssl.py", line 1309, in do_handshake2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: socket.timeout: _ssl.c:1114: The handshake operation timed out2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: During handling of the above exception, another exception occurred:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: Traceback (most recent call last):2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downloader.py", line 374, in _openfromurl2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: return self._retry(self._open_for_download, self.options['retry'])2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downnnnnnloader.py", line 224, in _retry2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: _checkRetry(retry, retries, e)2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downloader.py", line 208, in _checkRetry2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: raise exception2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downloader.py", line 214, in _retry2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: return func()2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downloader.py", line 264, in _open_for_download2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: robj = self._urlopen(self.url)2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downloader.py", line 198, in _urlopen2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: return opener.open(url, timeout=self.options['timeout'])2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/urllib/request.py", line 525, in open2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/urllib/request.py", line 542, in _open2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/urllib/request.py", line 502, in _call_chain2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/urllib/request.py", line 1397, in https_open2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/urllib/request.py", line 1357, in do_open2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: urllib.error.URLError: <urlopen error _ssl.c:1114: The handshake operation timed out>2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: During handling of the above exception, another exception occurred:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: Traceback (most recent call last):2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 881, in _getVibsFromUrls2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: with d.Open() as rObj:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downloader.py", line 473, in Open2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: return self._openfromurl()2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Downloader.py", line 376, in _openfromurl2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: raise DownloaderError(self.url, self.local, str(e))2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: vmware.esximage.Downloader.DownloaderError: ('https://VR:443/vib/vmware-hbr-agent.vib', '', '<urlopen error _ssl.c:1114: The handshake operation timed out>')2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: During handling of the above exception, another exception occurred:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR:2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: Traceback (most recent call last):2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 222, in main2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: cmd.Run()2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 148, in Run2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 2215, in InitVibInstall2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: Transaction.GetVibsAndMetaFromSources(vibUrls, metaUrls, vibSpecs,2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 250, in GetVibsAndMetaFromSources2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: vibsToInstall = Transaction._getVibsFromUrls(vibUrls)2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 905, in _getVibsFromUrls2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: raise Errors.VibDownloadError(url, None, str(e))2024-11-13T07:03:07Z esxupdate: 5076018: esxupdate: ERROR: vmware.esximage.Errors.VibDownloadError: ('https://VR:443/vib/vmware-hbr-agent.vib', None, "('https://VR:443/vib/vmware-hbr-agent.vib', '', '<urlopen error _ssl.c:1114: The handshake operation timed out>')") - vCenter has following error confirming host throwing error while installing VIB:
In/var/log/vmware/vpxd/vpxd.log:-- > (vim. fault.PlatformConfigFault) {faultCause = (vmodl.MethodFault) null,faultMessage = <unset>,text = "/usr/sbin/esxupdate returned with exit status: 7"msg = "An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 7."
Environment
VCenter : 7.0.3
vSphere replication: 9.0.2
Cause
- The inability to retrieve certificate details further suggests that there is a network issue or configuration problem with VR connectivity on port 443, preventing the handshake from completing.
- Issue could also been seen in an instance where port 443 needed for ESXi host to download
vmware-hbr-agentvib from vSphere Replication Appliance is blocked.
Resolution
1. Validate Port 443 Connectivity ( (Intra-Site Validation)
Confirm that port 443 is open between the ESXi host and the vSphere Replication appliance on the same site.
nc -zv <vSphere_Replication_IP> 443
Note: While the above command may indicate a connection, further validation is necessary to ensure proper communication
2. Validate by Installing HBR VIB on ESXi:
On the ESXi host, run the following command to verify communication
esxcli software vib install -v https://<vSphere_Replication_IP>:443/vib/vmware-hbr-agent.vib
If the firewall rule between the ESXi host and the vSphere Replication appliance is not properly configured, this command will timeout.
Firewall Rule Validation:
- Review the firewall settings to ensure port 443 is open from the ESXi host to the local vSphere Replication appliance (When it is Legacy replication)
- Add or update firewall rules as necessary to allow uninterrupted communication on port 443.
Additional Information
Run this command in vSphere Replication to confirm it is good
LD_LIBRARY_PATH=/usr/lib /usr/bin/wget --no-check-certificate https://VSpherereplication IP:443/vib/vmware-hbr-agent.vib
admin@dcvsra [ ~ ]$ LD_LIBRARY_PATH=/usr/lib /usr/bin/wget --no-check-certificate https://VSpherereplication:443/vib/vmware-hbr-agent.vib
--2024-11-19 10:49:41-- https://VR/vib/vmware-hbr-agent.vib
Connecting to VSphere replication IP:443... connected.
WARNING: cannot verify VSphere replication's certificate, issued by ‘C=US,ST=California,L=Palo Alto,OU=SRM,O=VMware\\, Inc.’:
Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 27407470 (26M) [text/plain]
Saving to: ‘vmware-hbr-agent.vib.1’
vmware-hbr-agent.vib.1 100%[========================================================================================================================================>] 26.14M --.-KB/s in 0.05s
Feedback
Yes
No
Powered by
