ESXi Root user lost access on SSH, Host Client, and Host Console after applying Desired state file.
search cancel

ESXi Root user lost access on SSH, Host Client, and Host Console after applying Desired state file.

book

Article ID: 422863

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

- Unable to login to ESXi SSH, Host Client, and Console with "root" user

- Lockdown mode is disabled

- root user is not expired but lost permissions

/var/run/log/hostd.log

YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101265]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<5xxxxxx8-7xx9-7xxc-4xxb-4xxxxxxxxxxx9, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 61276'>>, ha-sessionmgr, vim.SessionManager.login, <vim.version.version9, official, 5.5>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE]>
YYYY-MM-DDTHH:MM:SS.633Z Db(167) Hostd[2101265]: [Originator@6876 sub=Solo.Vmomi] Arg userName:
YYYY-MM-DDTHH:MM:SS.633Z Db(167) Hostd[2101224]: --> "local-root"
YYYY-MM-DDTHH:MM:SS.633Z Db(167) Hostd[2101265]: [Originator@6876 sub=Solo.Vmomi] Arg password:
YYYY-MM-DDTHH:MM:SS.633Z Db(167) Hostd[2101224]: --> (not shown)
YYYY-MM-DDTHH:MM:SS.633Z Db(167) Hostd[2101224]: -->
YYYY-MM-DDTHH:MM:SS.633Z Db(167) Hostd[2101265]: [Originator@6876 sub=Solo.Vmomi] Arg locale:
YYYY-MM-DDTHH:MM:SS.633Z Db(167) Hostd[2101224]: --> (null)
YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101265]: [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.NoPermission
YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101265]: [Originator@6876 sub=Solo.Vmomi] Result:
YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101224]: --> (vim.fault.NoPermission) {
YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101224]: --> object = 'vim.Folder:ha-folder-root',
YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101224]: --> privilegeId = "System.View",
YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101224]: --> msg = "",
YYYY-MM-DDTHH:MM:SS.633Z In(166) Hostd[2101224]: --> }

Environment

ESXi 8.x

ESXi 9.x

Cause

Desired State File has configuration issue that caused to delete permissions on the Full Admin users such as "root"

/var/run/log/settingsd.log

YYYY-MM-DDTHH:MM:SS.403Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] DELETE drift detected at /esx/authorization/permissions/0
YYYY-MM-DDTHH:MM:SS.403Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] DELETE drift detected at /esx/authorization/permissions/1
YYYY-MM-DDTHH:MM:SS.403Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] DELETE drift detected at /esx/authorization/permissions/2
YYYY-MM-DDTHH:MM:SS.403Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] DELETE drift detected at /esx/authorization/lockdown_settings

...

YYYY-MM-DDTHH:MM:SS.913Z In(14) settingsd[2101522]: debug [ConfigStore:609d46f640] Found match for dependency REMEDIATE:permissions
YYYY-MM-DDTHH:MM:SS.915Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] Plugin Launcher created for id=permissions, operation=APPLY, version=3, type=python
YYYY-MM-DDTHH:MM:SS.917Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] Forking plugin for id=permissions
YYYY-MM-DDTHH:MM:SS.917Z In(14) settingsd[2101522]: info [ConfigStore:609d4f0640] Starting plugin monitor thread
YYYY-MM-DDTHH:MM:SS.216Z In(14) settingsd[2101522]: info [ConfigStore:609d4f0640] Stopping plugin monitor thread
YYYY-MM-DDTHH:MM:SS.216Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] Config Manager plugin=permissions, finished successfully
YYYY-MM-DDTHH:MM:SS.217Z In(14) settingsd[2101522]: info [ConfigStore:609d46f640] Plugin permissions completed operation APPLY in 0.301551 seconds.
YYYY-MM-DDTHH:MM:SS.217Z Er(11) settingsd[2101522]: error [ConfigStore:609d46f640] permissions plugin failed to execute.

Resolution

- The issue is caused most possible Permissions missing under "ESXi Authorization" in Desired state file.

  • Add Permissions under "ESXi Authorization" in the Desired State Document and re-apply should fix the issue.

- If the issue persists, re-apply the original document, original config document available before the changes done when exported.

- Revert the Configuration of ESXi Host, which would be re-install of ESXi host

  • Reboot host > Shift+R to revert back

Workaround:

- Login with any existing/working Admin user (if available) to Host Client and manually add Admin role to "root" user

  • Or login with SSH as any existing/working Admin user (if available) and execute the following command in the shell: 
    • $ esxcli system permission set -i root -r Admin

- However,  customer needs to fix the desired state document and apply again.

Powered by Wolken Software