User [email protected] logged in as hbr-agent / hbrsrv messages are filling up host event logs

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

1. hbr-agent / hbrsrv login and log out messages are seen in the host event logs and hostd logs
2. An increase in such event logging or flood of messages can mask other important log messages in the host
3. Repetitive logins causes the vCenter DB to grow faster and eventually fill-up the database

 

Host Event logs:

MM/DD/YYYY, hh:mm:ss AM User [email protected] logged in as hbr-agent/7.0.2-17955482
MM/DD/YYYY, hh:mm:ss AM User [email protected] logged out (login time: Monday, January 10, 2022 10:17:48 AM EST, number of API invocations: 0, user agent: hbr-agent/7.0.2-17955482)

 

 

Environment

VMware vSphere Replication 8.x

Cause

vSphere Replication software requires hbr-agent or hbrsrv to query information from ESXi, such as VM or network configuration every 60 seconds. So for each host, there will be 1440 login events and 1440 logout events every day. This process only uses the 'root' user to perform this activity. 

Resolution

Currently, there is no resolution for this problem. 

Workaround:

STEP 1:

root@#####[ / ]# cd /opt/vmware/hms/conf/

root@#####[ /opt/vmware/hms/conf ]#

Edit hms-configuration.xml using vi and update the following entry from true to false.

<!-- Whether to install hbragent vib automatically when reconnecting to the hosts -->

<hms-auto-install-hbragent-vib>true</hms-auto-install-hbragent-vib>

NOTE: If replication traffic encryption is not used, we can change hms-auto-install-hbragent-vib to false in hms-configuration.xml and restart hms service. Then hms will not push the VIB to ESXi. Setting hms-auto-install-hbragent-vib to false can prevent installing it again. You must also uninstall the VIB or stop the hbrsrv service in the ESXi host.

In previous version before 8.x the service is also known as hbr-agent

STEP 2:

Restart HMS by running the command : systemctl restart hms

STEP 3:

Uninstall vmware-hbrsrv VIB from all the hosts.

[root@ESXi67APR:~] esxcli software vib list | grep -i hbr

The output should match the format shown below:
vmware-hbrsrv 8.0.3-0.0.24022510 VMware VMwareCertified YY-DD-MM host

To remove the VIB, run below command:
esxcli software vib remove -n vmware-hbr-agent

#You can also run a PowerCLI script to uninstall this agent from all the hosts in vCenter but before you run this script, please execute STEP 1 & 2.

CAUTION: You won't able to configure vSphere replication encryption on VMs, if you disable hbrsrv or uninstall the VIB. 

For a large infrastructure, please use the PowerCLI script below to uninstall hbr-agent vib from all the hosts in vCenter.

$VC = Read-Host 'Please enter VCenter Server Name'
Connect-VIServer $VC
$VIB = @{"vibname" = "vmware-hbr-agent"}
$Clusters = Get-Cluster
ForEach ($Cluster in $Clusters){
Write-output $Cluster.Name
$hosts = Get-Cluster $Cluster | Get-VMHost
forEach ($vihost in $hosts)
{
write-host $vihost
$esxcli = Get-EsxCli -vmhost $vihost -V2
$vibexists = $esxcli.software.vib.list.Invoke() | Where { $_.Name -like "*hbr-agent*"} | Select @{N="VMHost";E={$ESXCLI.VMHost}}, Name, Version
if ($vibexists -ne $null) {
Write-Output $vibexists
$esxcli.software.vib.remove.Invoke($VIB)
}
else {Write-Output 'VIB Not Present'}
}} 

Additional Information

Enabling lockdown mode on ESXi 7.0 U2 will result in generating events, Cannot login user [email protected]: no permission (312787)
ESX/ESXi reports in Task & Events: User [email protected] logged in (320909)