An error occurred during host configuration - Host fails to download vmware-hbr-agent VIB (Temporary failure in name resolution)

Products

• VMware Live Recovery

Issue/Introduction

Symptom:

• Following prompt with description "Apply Solution" will show below frequently running task in vCenter

 

1. Host fails to download vmware-hbr-agent.vib

2. Host constantly attempts to install vmware-hbr-agent.vib

You see the below tasks getting logged in ESXi - 

Install
esxi01.###.com
An error occurred during host configuration.
com.vmware.vr-sa-######-########-########-########@vsphere.local
dd/mm/yyyy, hh:mm:ss 
dd/mm/yyyy, hh:mm:ss 

Task Name: Install
Status   : An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 7.
Initiator: com.vmware.vr-######-########-########-#########@vsphere.local
Target   : Host 192.#.#.# 
Server   : vCenter vcsa.###.com

• You can also see Install error with "An error occurred during host configuration" on the vCenter.
  

Validation:

• "hbr-agent" service is missing on the vCenter. Select host > Configure > Services.
  
  
  
• SSH to the host where you see the install error and navigate to "/var/log/vpxa.log", to see below events -

yyyy-mm-ddThh:mm:ssZ info vpxa[2100734] [Originator@6876 sub=Default opID=######-########-########-########-HMSINT-29-72-65] [VpxLRO] -- ERROR task-41013 -- patchManager -- vim.host.PatchManager.InstallV2: vim.fault.PlatformConfigFault:
--> Result:
--> (vim.fault.PlatformConfigFault) {
-->  faultCause = (vmodl.MethodFault) null,
-->  faultMessage = <unset>,
-->  text = "/usr/sbin/esxupdate returned with exit status: 7"
-->  msg = "An error occurred during host configuration."
--> }
--> Args:
-->
--> Arg metaUrls:
--> (string) []
--> Arg bundleUrls:
--> (string) []
--> Arg vibUrls:
--> (string) [
-->  "https://vSphere-replication.###.com:8043/vib/vmware-hbr-agent.vib"
--> ]
--> Arg spec:
--> (vim.host.PatchManager.PatchManagerOperationSpec) {
-->  proxy = "http://vcenter.###.com",
-->  port = 80,
-->  userName = <unset>,
-->  password = <unset>,
-->  cmdOption = <unset>
--> }

• SSH to the vCenter where you see the install error and navigate to "/var/log/vmware/vpxd.log", to see below events -

yyyy-mm-ddThh:mm:ssZ info vpxd[07995] [Originator@6876 sub=Default opID=######-########-########-########HMSINT-46-18] [VpxLRO] -- ERROR task-23603 -- patchManager-10 -- vim.host.PatchManager.InstallV2: vim.fault.PlatformConfigFault:
--> Result:
--> (vim.fault.PlatformConfigFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    text = "/usr/sbin/esxupdate returned with exit status: 7"
-->    msg = "An error occurred during host configuration."
--> }
--> Args:
-->
--> Arg metaUrls:
--> (string) []
--> Arg bundleUrls:
--> (string) []
--> Arg vibUrls:
--> (string) [
-->    "https://replication.###.com:8043/vib/vmware-hbr-agent.vib"
--> ]
--> Arg spec:
--> (vim.host.PatchManager.PatchManagerOperationSpec) {
-->    proxy = "http://vCenter.###.com",
-->    port = 80,
-->    userName = <unset>,
-->    password = <unset>,
-->    cmdOption = <unset>
--> }

• SSH to the host where you see the install error and navigate to "/var/log/esxupdate.log", to see below events -

An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 7.

yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: esximage.Downloader.DownloaderError: ('https://192.###.###.:x8043/vib/vmware-hbr-agent.vib', '', '<urlopen error [Errno -3] Temporary failure in name resolution>')
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: During handling of the above exception, another exception occurred:
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: Traceback (most recent call last):
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: File "/usr/sbin/esxupdate", line 378, in main
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: cmd.Run()
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 161, in Run
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: res = t.InstallVibsFromSources(viburls, metaurls, None,
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 834, in InstallVibsFromSources
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: self.GetVibsAndMetaFromSources(vibUrls, metaUrls, vibspecs,
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: vibsToInstall = Transaction._getVibsFromUrls(vibUrls)
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: raise Errors.VibDownloadError(url, None, str(e))
yyyy-mm-ddThh:mm:ssZ Er(11) esxupdate[8059525]: esximage.Errors.VibDownloadError: ('https://192.###.###.###:8043/vib/vmware-hbr-agent.vib', None, "('https://192.###.###.###:8043/vib/vmware-hbr-agent.vib', '', '<urlopen error [Errno -3] Temporary failure in name resolution
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: esxupdate: INFO: --- Command: update Args: ['update'] Options: 
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: esxupdate: INFO: viburls = ['https://replication.###.com:8043/vib/vmware-hbr-agent.vib'] 
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: downloader: INFO: Opening https://replication.###.com:8043/vib/vmware-hbr-agent.vib for download 
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: downloader: INFO: Proxies configured: {'http': 'http://vCenter.###.com:80', 'https': 'http://vCenter.###.com:80', 'ftp': 'http://vCenter.###.com:80'} 
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: Proxy: ERROR: Proxy server cannot connect to the endpoint: HTTP/1.1 404 Not Found Connection: close Content-Type: text/html Content-Length: 48  
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: downloader: WARNING: Download failed: Proxy server vCenter.###.com:80 returned status 404, 4 retry left... yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: downloader: INFO: Proxies configured: {'http': 'http://vCenter.###.com:80', 'https': 'http://vCenter.###.com:80', 'ftp': 'http://vCenter.###.com:80'} 
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: Proxy: ERROR: Proxy server cannot connect to the endpoint: HTTP/1.1 404 Not Found Date: Connection: close Content-Type: text/html Content-Length: 48 
yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: downloader: WARNING: Download failed: Proxy server vCenter.###.com:80 returned status 404, 3 retry left... yyyy-mm-ddThh:mm:ssZ esxupdate: 2405325: downloader: INFO: Proxies configured: {'http': 'http://vCenter.###.com:80', 'https': 'http://vCenter.###.com:80', 'ftp': 'http://vCenter.###.com:80'} 

Environment

VMware vSphere Replication 8.x

VMware vSphere Replication 9.x

Cause

• DNS configuration is incorrect or not configured at all. Network ports are blocked on the firewall.

Resolution

• Check if the VIB is installed by running below command -

[root@ESXihost:~] esxcli software vib list | grep -i hbr
vmware-hbr-agent               8.0.2-0.0.22517424                     VMware  VMwareCertified   2024-03-21 

• Remove the VIB from the problematic host, if the vib is seen. 

[root@ESXihost:~] esxcli software vib remove -n vmware-hbr-agent
Removal Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed:
   VIBs Removed: VMware_bootbank_vmware-hbr-agent_8.0.2-0.0.22517424
   VIBs Skipped: 

• Try installing the VIB manually and check the output. Disable the firewall before performing this step and enable it after its complete. 
  

  esxcli network firewall set --enabled false
  esxcli network firewall set --enabled true (Set it back to true after downloading the file) 
  

•  Command to install the VIB manually 

[root@ESXihost:~] esxcli software vib install --proxy=http://vCenter.###.com:80 -v http://replication.###.com:8043/vib/vmware-hbr-agent.vib
 [VibDownloadError]
 ('https://replication.###.com:8043/vib/vmware-hbr-agent.vib', '', 'Proxy server vCenter.###.com:80 returned status 404')
        url = //http://replication.###.com:8043/vib/vmware-hbr-agent.vib
Please refer to the log file for more details. (Refer to esxupdate.log)

• Check to see if the VIB can be downloaded without any errors - http://replication.###.com:8043/vib/vmware-hbr-agent.vib
  
  

          Incase the above steps don't work then follow below steps -

      

• • Copy the file to the ESXi Host either using Winscp or Linux SCP command -
  • Download the vib file manually from the VR appliance by navigating to /opt/vmware/hbr/vib/ using winscp and download the file to local system. 

• • Upload the vib file to the local datastore of the host.

                       OR

• • On the vSphere Replication appliance find the vmware-hbr-agent vib using below commands-

                root [ /home/admin ]# find / -iname *hbr*vib*
                /opt/vmware/hbr/vib/vmware-hbr-agent-8.0.3-0.0.24299508.i386.vib
                /opt/vmware/hbr/vib/vmware-hbr-agent.vib

• • Once the vib has been identified copy it from the vSphere Replication appliance to the ESXi Host using the Linux SCP command.

                  root [ /home/admin ]# scp /opt/vmware/hbr/vib/vmware-hbr-agent-8.0.3-0.0.24299508.i386.vib root@Target ESXi Host IP address:/tmp
                 The authenticity of host '10.#.#.# (10.#.#.# )' can't be established.
                 ECDSA key fingerprint is SHA256:
                This key is not known by any other names
                 Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
                 Warning: Permanently added '10.#.#.# ' (ECDSA) to the list of known hosts.
                 (root@10.#.#.# ) Password:
                  vmware-hbr-agent-8.0.3-0.0.24299508.i386.vib                        100%   26MB 154.8MB/s   00:00

                 Post which install the vib file by running "esxcli software vib install -v <path to vib file>".

                 esxcli software vib install -v /tmp/vmware-hbr-agent-8.0.3-0.0.24299508.i386.vib
                 Installation Result
                 Message: Operation finished successfully.
                 VIBs Installed: VMware_bootbank_vmware-hbr-agent_8.0.3-0.0.24299508
                 VIBs Removed:
                 VIBs Skipped:
                 Reboot Required: false
                 DPU Results:

 

• Note:

  • Ensure that no nodes have duplicate hostnames in vCenter inventory 

  Bad Example:
  
  [root@vCenter:~] hostname
  vCenter.###.com
  
  [root@Atlanta:~] hostname -i
  172.#.#.# 
  
  root@ESXihost [ ~ ]# hostname
  vCenter.###.com
  
  root@ESXihost [ ~ ]# hostname -i
  127.#.#.# 

  - The host and vCenter have the same hostnames - vCenter.x.com 

  [root@localhost:~] hostname
  localhost.localdomain
  
  [root@localhost:~] hostname -i
  127.#.#.# 

  - Here, there is no DNS record created for this host. 

           
  DNS troubleshooting

  1. Make sure vCenter, vSphere Replication, SRM and ESXi hosts are using the correct DNS server. (Decommissioned DNS must not be used)

  2. Create forward and reverse lookup records in DNS for vCenter, vSphere Replication, SRM and ESXi hosts, if it does not exist. 

  3. If step 1 & 2 are not working, update the "hosts" file of vCenter, vSphere Replication and/or SRM.

   - Update the hosts file at "/etc/hosts" file in the format of "IP FQDN Hostname"; Example: "192.#.#.#  SRM01.###.local SRM01"

  1. Verify the DNS record for the host is correctly configured in DNS manager (forward/reverse lookup must be configured)
  
  2. Verify the hostname in DCUI matches the DNS record 
  
  3. Verify the Primary & alternate DNS servers are correct & working from the host.
  
  4. Verify the custom DNS suffix is set