ESXi Host Displays Repeated 'Cannot Login User [email protected]: No Permission' Events After Upgrading to Version 8.0 U3 with Lockdown Mode Enabled

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

After updating the ESXi host to 8.0 U3 with lockdown mode enabled, if the 'root' user is not added to the exception users list, the host starts triggering the event -

Type: Error
User: root
Target: ESXi.host.local
Description: Cannot login user [email protected]: no permission
Event Type Description: A user could not log in due to insufficient access permission

• Before enabling lockdown mode we see the following in the event viewer within VC server: "User [email protected] logged in as VMware-client/8.0.3", current date and time, "(i) Information", such as:

• After enabling Lockdown mode we see the following repeate every 5 minutes in the event viewer within VC server: "Cannot login user [email protected]: no permission", current date and time, "(!) Error", such as:

Within the ESXi host syslog.log, there will be entries similar to the following:

YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<52687575-9d5b-c00e-1e7d-1c2d6ed5ad1e, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 35710'>>, ha-sessionmgr, vim.SessionManager.login, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x0000002569ef9548]>
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Arg userName:
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: --> "local-root"
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Arg password:
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: --> (not shown)
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: -->
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Arg locale:
YYYY-MM-DDTHH:MM:SS.845Z Db(167) Hostd[#######]: --> "en"
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.NoPermission
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi] Result:
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> (vim.fault.NoPermission) {
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> object = 'vim.Folder:ha-folder-root',
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> privilegeId = "System.View",
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> msg = "",
YYYY-MM-DDTHH:MM:SS.845Z In(166) Hostd[#######]: --> }

When grepping for "Cannot login user [email protected]: no permission" from hostd.log, the error repeats every 5 minutes.

YYYY-MM-DDTHH:MM:SS.###Z In(166) Hostd[#######]: [Originator@#### sub=Vimsvc.ha-eventmgr opID=######## sid=########] Event ##### : Cannot login user [email protected]: no permission

The errors in hostd.log occur at the same time as the healthd.log entires below, which are also occurring every 5 minutes.

YYYY-MM-DDTHH:MM:SS.###Z In(166) healthd[#######]: [Originator@#### sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd

Additional symptoms reported

• You experience an unexpected ESXi host crash; the host goes offline and requires a reboot to bring it back online.
• You attach the host log bundle for review to determine the root cause and whether the issue is hardware or software related.
• During log review of an unexplained host offline event, you notice repeated "Cannot login user [email protected]: no permission" entries and want to confirm whether they are connected to the outage.

 

Environment

• VMware vSphere ESXi 8.0 U3 and newer

Cause

vsan_health is a plugin launched every 5 minutes to capture the vSANmgmt daemon health.
The plugin tries to log in to Hostd as a root user to get vSAN stubs.

When the host is in lockdown mode, the "root" user will be disabled.

Hence the error log is expected in hostd.log.

Note: The vsan_health plugin is used by components other then vSAN. As a result, this issue can be seen in non-VSAN environments. 

Additional Scenario: The error logging can be caused by daemons (from custom images / third party images)  using the root user for host operations, when in lockdown mode. This scenario can occur on ESXi versions later than ESXi 8.0u3e.

Resolution

This issue is resolved in ESXi 8.0u3e

Workaround 

1. Disable the vSAN health feature on the host, by running this command:

   # configstorecli config current set -c esx -g health -k vsan_health --path "enabled" --value False

2. To apply the change, restart the healthd service with:

   /etc/init.d/health restart

Additional Scenario: If this issue is caused by daemons (from custom images / third party images)  using the root user for host operations, when in lockdown mode. This can be prevented by stopping the daemon with the following command.

1. Open SSH to the ESXi, and run the command below to stop the daemon

2. /etc/init.d/<daemon> stop 

   1. replace "daemon" with the name of the daemon using the root user
   2. Please note, once the host is rebooted, the daemon will resume using the root user again.