Enabling Lockdown Mode on ESXi shows - Cannot login user root@127.0.0.1: no permission events
search cancel

Enabling Lockdown Mode on ESXi shows - Cannot login user [email protected]: no permission events

book

Article ID: 312787

calendar_today

Updated On:

Products

VMware Live Recovery VMware vSphere ESXi

Issue/Introduction

When the 'root' user is not added to the exception users list, the host starts triggering the event - 

Date Time:  <YYYY-MM-DD>, HH:MM:SS
Type: Error
User: root
Target:  ESXi.host.local
Description: Cannot login user [email protected]: no permission
Event Type Description: A user could not log in due to insufficient access permission
Possible Causes: 
The user account has insufficient access permission Action: Log in with a user account that has the necessary access permissions or grant additional access permissions to the current user

Before adding 'root' user to Exception List:

<YYYY-MM-DD>T<time> info hostd[2100450] [Originator@6876 sub=Default opID=########] Accepted password for user root from 127.0.0.1
<YYYY-MM-DD>T<time> info hostd[2100450] [Originator@6876 sub=Vimsvc opID=########] [Auth]: User root
<YYYY-MM-DD>T<time> warning hostd[2100450] [Originator@6876 sub=Vimsvc opID=########] Refresh function is not configured.User data can't be added to scheduler.User name: root
<YYYY-MM-DD>T<time> info hostd[2100450] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=########] Event 16503 : Cannot login user [email protected]: no permission
<YYYY-MM-DD>T<time> info hostd[2100451] [Originator@6876 sub=Vimsvc. TaskManager opID=sps-Main-######-###-
bb1e-d63f09da9491] Task Created : haTask -- vim.vslm.host.CatalogSyncManager.queryCatalogChange-##########
(YYYY-MM-DD>T<time> info hostd[2100454] [Originator@6876 sub=Libs opID=sps-Main-######-###-######-#-##-#### user=vpxuser:VSPHERE. LOCALvpxd-extension-######
############] notFound(403)
<YYYY-MM-DD>T<time> info hostd[2100454] [Originator@6876 sub=Libs opID=sps-Main-####
############] New error before the previous is handled
<YYYY-MM-DD>T<time> info hostd[2100454] [Originator@6876 sub=Vimsvc. TaskManager opID=sps-Main-######-###
####-#
<YYYY-MM-DD>T<time> info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Activation finished; << ####
127.0.0.1 : 21957'>>, ha-sessionmgr, vim. SessionManager. login>
<YYYY-MM-DD>T<time> verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Arg userName:
-- > "local-root"
<YYYY-MM-DD>T<time> verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Arg password:
-- > (not shown)
-- >
<YYYY-MM-DD>T<time> verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Arg locale:
-- > (null)
<YYYY-MM-DD>T<time> info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Throw vim. fault.NoPermission
<YYYY-MM-DD>T<time> info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Result:
-- > (vim.fault.NoPermission) {
-- > object = 'vim. Folder:ha-folder-root',
-- > privilegeId = "System.View",
-> msg = "",
->}

After adding 'root' user to Exception List:

<YYYY-MM-DD>T<time> info hostd[2102861] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=########] Event 16507 : User [email protected] logged in as hbr-agent/7.0.3-20217181

This issue has been reproduced with vCenter 8.0.2 and VRMS 8.8.0.2. When ESXi host is in lockdown mode, ESXi shows the event "Cannot login user [email protected]: no permission" every 1 minute.

less hbr-agent.log | grep -i 'Create login request for user local-root'  (In the ESXi host)

<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5254da700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d52555b700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root

hbr agent connects to the host repeatedly:

<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Connected
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create acquire local ticket request
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Write request
'YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Read response
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] HTTP 1/1 200 response
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] Got 6 HTTP headers
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] Content length: 558
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Transfer content: 0 bytes (558already in buffer)
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] debug: [AsyncVmomiClient] Acquired local ticket, logging
in ...
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Write request
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Read response
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d52555b700] error: [Http] Unexpected HTTP status code: 500
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Close connection
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d52555b700] trace: [HostdVmomiHttp] Clear session cookies
YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Connection closed
YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d525459700] trace: [Server] Removed 0 dead connections
YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: <YYYY-MM-DD>T<time> hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Connected

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

When an ESXi host is in  Configuring and Managing Lockdown Mode on ESXi Hosts, you can use the Exception User list to  Specify Lockdown Mode Exception Users in the VMware Host Client   & add the accounts of third-party solutions and external applications that need to access the host directly when the host is in lockdown mode. 

vSphere Replication software requires hbr-agent to query information from ESXi, such as VM or network configuration every 60 seconds. So for each host, there will be 1440 login events and 1440 logout events every day. This process only uses the 'root' user to perform this activity. Now, due to the nature of ESXi lockdown mode and vSphere Replication products requiring to work in their own way to achieve the results of their own, it creates a catch-22 situation. 

Therefore, the ESXi host ends up generating the errors when 'root' user isn't added to the Exception Users list. 

Resolution

NOTE:  You won't be able to configure vSphere replication encryption on VMs, if you disable hbr-agent or uninstall it from the host. 

Disable hbr-agent/hbrsrv service from the host and set it to start & stop manually. This must be done on all the ESXi hosts that you choose to enable lockdown mode on. 
 
Under ESXi > Configure > Services > hbrsrv or hbr-agent 

 
After the service selection, select edit Startup Policy > select start and stop manually > OK.



Additional Information

User [email protected] logged in as hbr-agent messages are filling up host event logs (312758)

Impact/Risks:

The host will perpetually fill up with Cannot login user [email protected]: no permission events, thereby obstructing other important events from populating in the events tab. 

Powered by Wolken Software