User [email protected] logged in as hbr-agent messages are filling up host event logs

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

1. hbr-agent login and log out messages are seen in the host event logs and hostd logs
2. An increase in such event logging or flood of messages can mask other important log messages in the host
3. Repetitive logins causes the vCenter DB to grow faster and eventually fill-up the database

 

Host Event logs:

01/10/2022, 10:18:48 AM User [email protected] logged in as hbr-agent/7.0.2-17955482
01/10/2022, 10:17:48 AM User [email protected] logged out (login time: Monday, January 10, 2022 10:17:48 AM EST, number of API invocations: 0, user agent: hbr-agent/7.0.2-17955482)

Environment

VMware vSphere Replication 8.x

Cause

vSphere Replication software requires hbr-agent to query information from ESXi, such as VM or network configuration every 60 seconds. So for each host, there will be 1440 login events and 1440 logout events every day. This process only uses the 'root' user to perform this activity. 

Resolution

Currently, there is no resolution for this problem. 

Workaround:

STEP 1:

root@VRMSPR [ / ]# cd /opt/vmware/hms/conf/

root@VRMSPR [ /opt/vmware/hms/conf ]#

Edit hms-configuration.xml using vi and update the following entry from true to false.

<!-- Whether to install hbragent vib automatically when reconnecting to the hosts -->

<hms-auto-install-hbragent-vib>true</hms-auto-install-hbragent-vib>

NOTE: If replication traffic encryption is not used, we can change hms-auto-install-hbragent-vib to false in hms-configuration.xml and restart hms service. Then hms will not push the VIB to ESXi. Setting hms-auto-install-hbragent-vib to false can prevent installing it again. You must also uninstall the VIB or stop the hbr-agent service in the ESXi host.

STEP 2:

Restart HMS by running the command : systemctl restart hms

STEP 3:

Uninstall vmware-hbr-agent VIB from all the hosts.

[root@ESXi67APR:~] esxcli software vib list | grep -i hbr

vmware-hbr-agent 6.8.6-0.0.13337832 VMware VMwareCertified 2019-07-12

esxcli software vib remove -n vmware-hbr-agent

You can also run a PowerCLI script to uninstall this agent from all the hosts in vCenter but before you run this script, please execute STEP 1 & 2.

CAUTION: You won't able to configure vSphere replication encryption on VMs, if you disable hbr-agent or uninstall the VIB. 

For a large infrastructure, please use the PowerCLI script below to uninstall hbr-agent vib from all the hosts in vCenter.

$VC = Read-Host 'Please enter VCenter Server Name'
Connect-VIServer $VC
$VIB = @{"vibname" = "vmware-hbr-agent"}
$Clusters = Get-Cluster
ForEach ($Cluster in $Clusters){
Write-output $Cluster.Name
$hosts = Get-Cluster $Cluster | Get-VMHost
forEach ($vihost in $hosts)
{
write-host $vihost
$esxcli = Get-EsxCli -vmhost $vihost -V2
$vibexists = $esxcli.software.vib.list.Invoke() | Where { $_.Name -like "*hbr-agent*"} | Select @{N="VMHost";E={$ESXCLI.VMHost}}, Name, Version
if ($vibexists -ne $null) {
Write-Output $vibexists
$esxcli.software.vib.remove.Invoke($VIB)
}
else {Write-Output 'VIB Not Present'}
}} 

Additional Information

vCenter Server event log filled with "User root logged out" and "User [email protected] logged in" messages (2054511)
[Internal] Enabling the lockdown mode on the ESXi 7.0 U2 will result in generating events, Cannot login user [email protected]: no permission (85326)
ESX/ESXi reports in Task & Events: User [email protected] logged in (1031578)