ESXi Host Displays Repeated 'Cannot Login User [email protected]: No Permission' Events After Upgrading to Version 8.0 U3 with Lockdown Mode Enabled

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

After updating the ESXi host to 8.0 U3 with lockdown mode enabled, if the 'root' user is not added to the exception users list, the host starts triggering the event - 

Type: Error
User: root
Target:  ESXi.host.local
Description: Cannot login user [email protected]: no permission
Event Type Description: A user could not log in due to insufficient access permission

- Before enabling lockdown mode we see the below in the event viewer within VC server:

- After enabling Lockdown mode we see the below in the event viewer within VC server: These errors will repeat every 5 minutes. 

Within the ESXi host syslog.log, there will be entries similar to the following:
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<52687575-9d5b-c00e-1e7d-1c2d6ed5ad1e, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 35710'>>, ha-sessionmgr, vim.SessionManager.login, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x0000002569ef9548]>
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Arg userName:
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: --> "local-root"
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Arg password:
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: --> (not shown)
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: -->
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Arg locale:
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: --> "en"
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.NoPermission
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Result:
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> (vim.fault.NoPermission) {
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> object = 'vim.Folder:ha-folder-root',
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> privilegeId = "System.View",
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> msg = "",
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> }

When grepping for "Cannot login user [email protected]: no permission" from hostd.log, the error repeats every 5 minutes. 
YYYY-MM-DDTHH:MM:SS.###Z In(166) Hostd[#######]: [Originator@#### sub=Vimsvc.ha-eventmgr opID=######## sid=########] Event ##### : Cannot login user [email protected]: no permission

The errors in hostd.log occur at the same time as the healthd.log entires below, which are also occurring every 5 minutes. 
YYYY-MM-DDTHH:MM:SS.###Z In(166) healthd[#######]: [Originator@#### sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd

 

Environment

VMware vSphere ESXi 8.0 U3

VMware vSphere ESXi 8.0 U3 +

Cause

The vsan_health is a plugin launched every 5 minutes to capture the vSANmgmt daemon health.
The plugin tries to log in to Hostd as a root user to get vsan stubs.

When the host is in lockdown mode, the "root" user will be disabled.

Hence the error log is expected in hostd.log.

 

Note: The vsan_health plugin is used by components other then vSAN. As a result, this issue can be seen in non-VSAN environments. 

Additional Scenario: The error logging can be caused by daemons (from custom images / third party images)  using the root user for host operations, when in lockdown mode. This scenario can occur on ESXi versions later than ESXi 8.0u3e.

Resolution

This issue is resolved in ESXi 8.0u3e

Workaround 

1. Disable the vSAN health feature on the host, by running this command:

   # configstorecli config current set -c esx -g health -k vsan_health --path "enabled" --value False

2. To apply the change, restart the healthd service with:

   /etc/init.d/health restart

Additional Scenario: If this issue is caused by daemons (from custom images / third party images)  using the root user for host operations, when in lockdown mode. This can be prevented by stopping the daemon with the following command.

1. Open SSH to the ESXi, and run the command below to stop the daemon

2. /etc/init.d/<daemon> stop 

   1. replace "daemon" with the name of the daemon using the root user
   2. Please note, once the host is rebooted, the daemon will resume using the root user again.