Enabling Lockdown Mode on an ESXi host results in - Cannot login user [email protected]: no permission events

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When the 'root' user is not added to the exception users list, the host starts triggering the event - 

Date Time:  03/25/2024, 2:05:19 PM
Type: Error
User: root
Target:  ESXi.host.local
Description: Cannot login user [email protected]: no permission
Event Type Description: A user could not log in due to insufficient access permission
Possible Causes: 
The user account has insufficient access permission Action: Log in with a user account that has the necessary access permissions or grant additional access permissions to the current user

Before adding 'root' user to Exception List:

2024-03-18T18:26:03.317Z info hostd[2100450] [Originator@6876 sub=Default opID=b1306eaf] Accepted password for user root from 127.0.0.1
2024-03-18T18:26:03.317Z info hostd[2100450] [Originator@6876 sub=Vimsvc opID=b1306eaf] [Auth]: User root
2024-03-18T18:26:03.317Z warning hostd[2100450] [Originator@6876 sub=Vimsvc opID=b1306eaf] Refresh function is not configured.User data can't be added to scheduler.User name: root
2024-03-18T18:26:03.317Z info hostd[2100450] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=b1306eaf] Event 16503 : Cannot login user [email protected]: no permission
2024-03-18T18:26:03.937Z info hostd[2100451] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-584081-203-786363-b-2c-6eb2 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-ff65a043-ba8f-4dcf-bb1e-d63f09da9491] Task Created : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-2704596614
2024-03-18T18:26:03.942Z info hostd[2100454] [Originator@6876 sub=Libs opID=sps-Main-584081-203-786363-b-2c-6eb2 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-ff65a043-ba8f-4dcf-bb1e-d63f09da9491] notFound(403)
2024-03-18T18:26:03.942Z info hostd[2100454] [Originator@6876 sub=Libs opID=sps-Main-584081-203-786363-b-2c-6eb2 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-ff65a043-ba8f-4dcf-bb1e-d63f09da9491] New error before the previous is handled
2024-03-18T18:26:03.942Z info hostd[2100454] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-584081-203-786363-b-2c-6eb2 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-ff65a043-ba8f-4dcf-bb1e-d63f09da9491] Task Completed : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-2704596614 Status success
2024-03-18T18:26:06.320Z info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=b1306eaf] Activation finished; <<52639ae0-ae87-a6f4-1075-6fb1a2eaf73b, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 21957'>>, ha-sessionmgr, vim.SessionManager.login>
2024-03-18T18:26:06.320Z verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=b1306eaf] Arg userName:
--> "local-root"
2024-03-18T18:26:06.320Z verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=b1306eaf] Arg password:
--> (not shown)
-->
2024-03-18T18:26:06.320Z verbose hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=b1306eaf] Arg locale:
--> (null)
2024-03-18T18:26:06.320Z info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=b1306eaf] Throw vim.fault.NoPermission
2024-03-18T18:26:06.320Z info hostd[2099895] [Originator@6876 sub=Solo.Vmomi opID=b1306eaf] Result:
--> (vim.fault.NoPermission) {
-->  object = 'vim.Folder:ha-folder-root',
-->  privilegeId = "System.View",
-->  msg = "",
--> }

After adding 'root' user to Exception List:

2024-03-18T18:27:03.318Z info hostd[2102861] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=b1306efd] Event 16507 : User [email protected] logged in as hbr-agent/7.0.3-20217181

This issue has been reproduced with vCenter 8.0.2 and VRMS 8.8.0.2. When ESXi host is in lockdown mode, ESXi shows the event "Cannot login user [email protected]: no permission" every 1 minute.

less hbr-agent.log | grep -i 'Create login request for user local-root'  (In the ESXi host)

2024-04-15T09:10:18.554Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:10:18.554052 hbr-agent-bin [1060145] [0x000000d5254da700] trace: [HostdVmomiHttp] Create login request for user local-root
2024-04-15T09:11:18.553Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:11:18.553592 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
2024-04-15T09:12:18.553Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:12:18.553125 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
2024-04-15T09:13:18.553Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:13:18.553866 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root
2024-04-15T09:14:18.553Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:14:18.553382 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [HostdVmomiHttp] Create login request for user local-root
2024-04-15T09:15:18.554Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:15:18.554915 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create login request for user local-root

hbr agent connects to the host repeatedly:

2024-04-15T09:19:18.549Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549832 hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Connected
2024-04-15T09:19:18.549Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549900 hbr-agent-bin [1060145] [0x000000d525459700] trace: [HostdVmomiHttp] Create acquire local ticket request
2024-04-15T09:19:18.549Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549915 hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Write request
2024-04-15T09:19:18.549Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.549950 hbr-agent-bin [1060145] [0x000000d525459700] trace: [AsyncVmomiClient] Read response
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552711 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] HTTP 1/1 200 response
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552774 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] Got 6 HTTP headers
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552787 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [Http] Content length: 558
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552796 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Transfer content: 0 bytes (558 already in buffer)
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552900 hbr-agent-bin [1060145] [0x000000d5253d8700] debug: [AsyncVmomiClient] Acquired local ticket, logging in...
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552917 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [HostdVmomiHttp] Create login request for user local-root
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552939 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Write request
2024-04-15T09:19:18.552Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:18.552969 hbr-agent-bin [1060145] [0x000000d5253d8700] trace: [AsyncVmomiClient] Read response
2024-04-15T09:19:21.557Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557827 hbr-agent-bin [1060145] [0x000000d52555b700] error: [Http] Unexpected HTTP status code: 500
2024-04-15T09:19:21.557Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557873 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Close connection
2024-04-15T09:19:21.557Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557883 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [HostdVmomiHttp] Clear session cookies
2024-04-15T09:19:21.557Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:19:21.557931 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Connection closed
2024-04-15T09:20:17.583Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:20:17.583308 hbr-agent-bin [1060145] [0x000000d525459700] trace: [Server] Removed 0 dead connections
2024-04-15T09:20:18.550Z In(166) hbr-agent-bin[1060145]: 2024-04-15T09:20:18.550689 hbr-agent-bin [1060145] [0x000000d52555b700] trace: [AsyncVmomiClient] Connected

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 8.0
VMware vSphere 7.0.x

Cause

When an ESXi host is in Lockdown Mode , you can use the Exception User list to Specify Lockdown Mode Exception Users & add the accounts of third-party solutions and external applications that need to access the host directly when the host is in lockdown mode. 

vSphere Replication software requires hbr-agent to query information from ESXi, such as VM or network configuration every 60 seconds. So for each host, there will be 1440 login events and 1440 logout events every day. This process only uses the 'root' user to perform this activity. Now, due to the nature of ESXi lockdown mode and vSphere Replication products requiring to work in their own way to achieve the results of their own, it creates a catch-22 situation. 

Therefore, the ESXi host ends up generating the errors when 'root' user isn't added to the Exception Users list. 

Resolution

Currently, there is no resolution for this problem. 

Workaround:

NOTE: You won't be able to configure vSphere replication encryption on VMs, if you disable hbr-agent or uninstall it from the host. 

Disable hbr-agent service from the host and set it to start & stop manually. This must be done on all the ESXi hosts that you choose to enable lockdown mode on. 

Additional Information

User [email protected] logged in as hbr-agent messages are filling up host event logs (87700) - https://kb.vmware.com/s/article/87700?lang=en_US

Impact/Risks:

The host will perpetually fill up with Cannot login user [email protected]: no permission events, thereby obstructing other important events from populating in the events tab.