"Cannot login user [email protected]: no permission" in ESXi host events when lockdown mode is enabled on ESXi 8.0u3

Products

• VMware vSphere ESXi

Issue/Introduction

• The ESXi has lockdown mode enabled and post upgrade to 8.0U3, Events stating "Cannot login user [email protected]: no permission" is noticed every 5 mins in the ESXi.
  
  
• In /var/run/log/hostd.log, the following messages are noticed.
  [YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098727]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=ad545190 sid=52687575] Event 1488 : Cannot login user [email protected]: no permission
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<52687575-9d5b-c00e-1e7d-1c2d6ed5ad1e, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 35710'>>, ha-sessionmgr, vim.SessionManager.login, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x0000002569ef9548]>
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Arg userName:
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: --> "local-root"
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Arg password:
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: --> (not shown)
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: -->
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Arg locale:
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: --> "en"
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.NoPermission
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Result:
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098681]: --> (vim.fault.NoPermission)


• Upon checking we see a watchdog for healthd which uses root user.
  # cat /var/run/vmware/watchdog/healthd

PID=525564
NRESTART=0
MAXRESTART=1000000
TSTAMP=1725485057
NQUICK=0
TQUICK=60
MAXQUICK=5
IMMORTAL=0
ARG=/usr/lib/vmware/healthd/bin/healthd
ARG=++securitydom=healthdDom,group=healthd
ARG=-r
ARG=/var/lib/vmware/osdata/healthd
ENV=USER=root <========= Uses root user
ENV=SHLVL=2
ENV=HOME=/
ENV=TERM=vt102
ENV=PATH=/bin
ENV=SHELL=/bin/sh
ENV=PWD=/
RLIMITS=-1,524288,256,4096


• ps -cCJ command confirms that the healthd runs every 5 minutes. 
  
  To capture processes for every second in a text file, run the below command:
  # while true; do date >> /tmp/ps_CcJ.txt && ps -CcJ >> /tmp/ps_CcJ.txt; sleep 1; done

We can see the below entries in tmp/ps_CcJ.txt file in interval of 5 minutes.
  
  528033  528033  vsan_health                        528033  524451  /usr/lib/vmware/healthd/plugins/bin/vsan_health -u http://!vmwLocalSocketHealthd
  528034  528033  worker                             528033  524451  /usr/lib/vmware/healthd/plugins/bin/vsan_health -u http://!vmwLocalSocketHealthd
  528035  528033  worker                             528033  524451  /usr/lib/vmware/healthd/plugins/bin/vsan_health -u http://!vmwLocalSocketHealthd
  
  

Example output of the healthd.log on ESXi: 

2024-12-09T16:30:30.196Z In(166) healthd[2102511]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:30:30.370Z In(166) healthd[2102511]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:35:30.199Z In(166) healthd[2100119]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:35:30.368Z In(166) healthd[2102511]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:40:30.203Z In(166) healthd[2100130]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:40:30.369Z In(166) healthd[2102511]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:45:30.206Z In(166) healthd[2100119]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:45:30.373Z In(166) healthd[2100130]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:50:30.209Z In(166) healthd[2100114]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:50:30.374Z In(166) healthd[2100111]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:55:30.212Z In(166) healthd[2100114]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:55:30.397Z In(166) healthd[2100114]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus

 

Environment

VMware vSphere ESXi 8.0U3

Cause

vsan_health is a plugin launched every 5 minutes to capture the vSANmgmt daemon health. The plugin tries to log in to Hostd as a root user to get vsan stubs. When the host is in lockdown mode, the "root" user will be disabled. Hence we get the error in hostd.log

 

 

Resolution

Broadcom has addressed the issue and a fix is scheduled to be in a future release of ESXi 8.0.

 

Please follow the below workaround as a temporary correction until fix has been released.

Workaround:

1. Disable the vsan_health plugin on the host, via command:
   configstorecli config current set -c esx -g health -k vsan_health --path "enabled" --value False
   
   
2. Restart the healthd via command:
   /etc/init.d/health restart

Disabling this `vsan_health` plugin and restarting healthd, this error(s) should not come in hostd.log.

Additional Information