Enabling Lockdown Mode on ESXi shows - Cannot login user [email protected]: no permission events

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When the 'root' user is not added to the exception users list, the host starts triggering the event - 

Date Time:  <YYYY-MM-DD>, HH:MM:SS
Type: Error
User: root
Target:  ESXi.host.local
Description: Cannot login user [email protected]: no permission
Event Type Description: A user could not log in due to insufficient access permission
Possible Causes: 
The user account has insufficient access permission Action: Log in with a user account that has the necessary access permissions or grant additional access permissions to the current user

Before adding 'root' user to Exception List:

<YYYY-MM-DD>T<time> info hostd[2100450] [Originator@6876 sub=Default opID=########] Accepted password for user root from 127.0.0.1
<YYYY-MM-DD>T<time> info hostd[2100450] [Originator@6876 sub=Vimsvc opID=########] [Auth]: User root
<YYYY-MM-DD>T<time> warning hostd[2100450] [Originator@6876 sub=Vimsvc opID=########] Refresh function is not configured.User data can't be added to scheduler.User name: root
<YYYY-MM-DD>T<time> info hostd[2100450] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=########] Event 16503 : Cannot login user [email protected]: no permission
<YYYY-MM-DD>T<time> info hostd[2100451] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-######-###-######-#-##-#### user=vpxuser:VSPHERE.LOCAL\vpxd-extension-ff65a043-ba8f-4dcf-bb1e-d63f09da9491] Task Created : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-##########
<YYYY-MM-DD>T<time> info hostd[2100454] [Originator@6876 sub=Libs opID=sps-Main-######-###-######-#-##-#### user=vpxuser:VSPHERE.LOCAL\vpxd-extension-########-####-####-####-############] notFound(403)
<YYYY-MM-DD>T<time> info hostd[2100454] [Originator@6876 sub=Libs opID=sps-Main-######-###-######-#-##-#### user=vpxuser:VSPHERE.LOCAL\vpxd-extension-########-####-####-####-############] New error before the previous is handled
<YYYY-MM-DD>T<time> info hostd[2100454] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-######-###-######-#-##-#### user=vpxuser:VSPHERE.LOCAL\vpxd-extension-########-####-####-####-############] Task Completed : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-########## Status success
<YYYY-MM-DD>T<time> info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Activation finished; <<########-####-####-####-############, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 21957'>>, ha-sessionmgr, vim.SessionManager.login>
<YYYY-MM-DD>T<time> verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Arg userName:
--> "local-root"
<YYYY-MM-DD>T<time> verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Arg password:
--> (not shown)
-->
<YYYY-MM-DD>T<time> verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Arg locale:
--> (null)
<YYYY-MM-DD>T<time> info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Throw vim.fault.NoPermission
<YYYY-MM-DD>T<time> info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=########] Result:
--> (vim.fault.NoPermission) {
-->  object = 'vim.Folder:ha-folder-root',
-->  privilegeId = "System.View",
-->  msg = "",
--> }

After adding 'root' user to Exception List:

<YYYY-MM-DD>T<time> info hostd[2102861] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=b1306efd] Event 16507 : User [email protected] logged in as hbr-agent/7.0.3-20217181

This issue has been reproduced with vCenter 8.0.2 and VRMS 8.8.0.2. When ESXi host is in lockdown mode, ESXi shows the event "Cannot login user [email protected]: no permission" every 1 minute.

less hbr-agent.log | grep -i 'Create login request for user local-root'  (In the ESXi host)

<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:10:18.554052 hbr-agent-bin [1060145] [0x000000d5254da700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:11:18.553592 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:12:18.553125 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:13:18.553866 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:14:18.553382 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:15:18.554915 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root

hbr agent connects to the host repeatedly:

<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549832 hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Connected
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549900 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create acquire local ticket request
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549915 hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Write request
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549950 hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Read response
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552711 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] HTTP 1/1 200 response
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552774 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] Got 6 HTTP headers
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552787 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] Content length: 558
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552796 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Transfer content: 0 bytes (558 already in buffer)
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552900 hbr-agent-bin [1060145] [0x000000d5253d8700] debug: [AsyncVmomiClient] Acquired local ticket, logging in...
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552917 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [HostdVmomiHttp] Create login request for user local-root
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552939 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Write request
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552969 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Read response
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557827 hbr-agent-bin [1060145] [0x000000d52555b700] error: [Http] Unexpected HTTP status code: 500
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557873 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Close connection
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557883 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [HostdVmomiHttp] Clear session cookies
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557931 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Connection closed
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:20:17.583308 hbr-agent-bin [1060145] [0x000000d525459700] trace: [Server] Removed 0 dead connections
<YYYY-MM-DD>T<time> In(166) hbr-agent-bin[1060145]: 2024-04-15T09:20:18.550689 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Connected

 

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

When an ESXi host is in Lockdown Mode , you can use the Exception User list to Specify Lockdown Mode Exception Users & add the accounts of third-party solutions and external applications that need to access the host directly when the host is in lockdown mode. 

vSphere Replication software requires hbr-agent to query information from ESXi, such as VM or network configuration every 60 seconds. So for each host, there will be 1440 login events and 1440 logout events every day. This process only uses the 'root' user to perform this activity. Now, due to the nature of ESXi lockdown mode and vSphere Replication products requiring to work in their own way to achieve the results of their own, it creates a catch-22 situation. 

Therefore, the ESXi host ends up generating the errors when 'root' user isn't added to the Exception Users list. 

Resolution

NOTE: You won't be able to configure vSphere replication encryption on VMs, if you disable hbr-agent or uninstall it from the host. 

Disable hbr-agent service from the host and set it to start & stop manually. This must be done on all the ESXi hosts that you choose to enable lockdown mode on. 

Additional Information

User [email protected] logged in as hbr-agent messages are filling up host event logs (312758)

Impact/Risks:

The host will perpetually fill up with Cannot login user [email protected]: no permission events, thereby obstructing other important events from populating in the events tab.