"Cannot login user root@127.0.0.1: no permission" in ESXi host events when lockdown mode is enabled on ESXi 8.0u3
search cancel

"Cannot login user [email protected]: no permission" in ESXi host events when lockdown mode is enabled on ESXi 8.0u3

book

Article ID: 376992

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • The ESXi has lockdown mode enabled and post upgrade to 8.0U3, Events stating "Cannot login user [email protected]: no permission" is noticed every 5 mins in the ESXi.

  • In /var/run/log/hostd.log, the following messages are noticed.
    [YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098727]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=ad545190 sid=52687575] Event 1488 : Cannot login user [email protected]: no permission
    [YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<52687575-9d5b-c00e-1e7d-1c2d6ed5ad1e, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 35710'>>, ha-sessionmgr, vim.SessionManager.login, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x0000002569ef9548]>
    [YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Arg userName:
    [YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: --> "local-root"
    [YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Arg password:
    [YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: --> (not shown)
    [YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: -->
    [YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Arg locale:
    [YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2098681]: --> "en"
    [YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.NoPermission
    [YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098715]: [Originator@6876 sub=Solo.Vmomi] Result:
    [YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2098681]: --> (vim.fault.NoPermission)

  • Upon checking we see a watchdog for healthd which uses root user.
    # cat /var/run/vmware/watchdog/healthd

    PID=525564
    NRESTART=0
    MAXRESTART=1000000
    TSTAMP=1725485057
    NQUICK=0
    TQUICK=60
    MAXQUICK=5
    IMMORTAL=0
    ARG=/usr/lib/vmware/healthd/bin/healthd
    ARG=++securitydom=healthdDom,group=healthd
    ARG=-r
    ARG=/var/lib/vmware/osdata/healthd
    ENV=USER=root <========= Uses root user
    ENV=SHLVL=2
    ENV=HOME=/
    ENV=TERM=vt102
    ENV=PATH=/bin
    ENV=SHELL=/bin/sh
    ENV=PWD=/
    RLIMITS=-1,524288,256,4096

  • ps -cCJ command confirms that the healthd runs every 5 minutes. 

    To capture processes for every second in a text file, run the below command:
    # while true; do date >> /tmp/ps_CcJ.txt && ps -CcJ >> /tmp/ps_CcJ.txt; sleep 1; done

    We can see the below entries in tmp/ps_CcJ.txt file in interval of 5 minutes.

    528033  528033  vsan_health                        528033  524451  /usr/lib/vmware/healthd/plugins/bin/vsan_health -u http://!vmwLocalSocketHealthd
    528034  528033  worker                             528033  524451  /usr/lib/vmware/healthd/plugins/bin/vsan_health -u http://!vmwLocalSocketHealthd
    528035  528033  worker                             528033  524451  /usr/lib/vmware/healthd/plugins/bin/vsan_health -u http://!vmwLocalSocketHealthd

Example output of the healthd.log on ESXi: 

2024-12-09T16:30:30.196Z In(166) healthd[2102511]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:30:30.370Z In(166) healthd[2102511]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:35:30.199Z In(166) healthd[2100119]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:35:30.368Z In(166) healthd[2102511]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:40:30.203Z In(166) healthd[2100130]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:40:30.369Z In(166) healthd[2102511]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:45:30.206Z In(166) healthd[2100119]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:45:30.373Z In(166) healthd[2100130]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:50:30.209Z In(166) healthd[2100114]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:50:30.374Z In(166) healthd[2100111]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus
--
2024-12-09T16:55:30.212Z In(166) healthd[2100114]: [Originator@6876 sub=PluginLauncher] Launching binary: /usr/lib/vmware/healthd/plugins/bin/vsan_health ++group=healthd-plugins,mem=40 -u http://!vmwLocalSocketHealthd
2024-12-09T16:55:30.397Z In(166) healthd[2100114]: [Originator@6876 sub=HealthdHandler] Report Path: /var/lib/vmware/osdata/healthd/vmw.vSANStatus

 

Environment

VMware vSphere ESXi 8.0U3

Cause

vsan_health is a plugin launched every 5 minutes to capture the vSANmgmt daemon health. The plugin tries to log in to Hostd as a root user to get vsan stubs. When the host is in lockdown mode, the "root" user will be disabled. Hence we get the error in hostd.log

 

 

Resolution

Broadcom has addressed the issue and a fix is scheduled to be in a future release of ESXi 8.0.

 

Please follow the below workaround as a temporary correction until fix has been released.

Workaround:

  1. Disable the vsan_health plugin on the host, via command:
    configstorecli config current set -c esx -g health -k vsan_health --path "enabled" --value False

  2. Restart the healthd via command:
    /etc/init.d/health restart

Disabling this `vsan_health` plugin and restarting healthd, this error(s) should not come in hostd.log.

Additional Information