Enabling Lockdown mode on ESXi host returns error message "Operation Failed! A general system error occurred: Internal error"

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0

Issue/Introduction

Unable to enable lockdown mode on the host from Host Profile, PowerCLI, vSphere, ESXi Shell and DCUI.

Cause

 

Examples of the following logs show the issue:

/var/log/hostd.log

YYYY-MM-DDTHH:MM:SS.766Z info hostd[2100556] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-f7-d200] Event 5077 : User [email protected] logged in as VMware-client/6.5.0
YYYY-MM-DDTHH:MM:SS.771Z info hostd[2099941] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-f7-d204 user=dcui] Task Created : haTask-ha-host-vim.host.HostAccessManager.changeLockdownMode-123333759
YYYY-MM-DDTHH:MM:SS.782Z info hostd[2100550] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-f7-d204 user=dcui] Event 5078 : Permission rule removed for root on root
YYYY-MM-DDTHH:MM:SS.782Z warning hostd[2099935] [Originator@6876 sub=UserDirectory opID=vim-cmd-f7-d204 user=dcui] Group lookup failed for '{adgroupname}'
YYYY-MM-DDTHH:MM:SS.783Z error hostd[2099935] [Originator@6876 sub=Vimsvc.AuthorizationManager opID=vim-cmd-f7-d204 user=dcui] Cannot remove ACE: N7Vmacore9Authorize25AuthUserNotFoundExceptionE(Group adgroupname) --> [context]zKq7AVICAgAAAJXqSgETaG9zdGQAAGVDF2xpYnZtYWNvcmUuc28AARoLumhvc3RkAAE0b7sA0ycqAELGJwBvzicAFUooAC7DKAAxxigBhSq2AT2RtgHuS7qCYZUYAWxpYnZpbS10eXBlcy5zbwABSptSAMzHLQBUAy4AAhE/Azt9AGxpYnB0aHJlYWQuc28uMAAEbdEObGliYy5zby42AA==[/context]
YYYY-MM-DDTHH:MM:SS.789Z error hostd[2099935] [Originator@6876 sub=Vimsvc.AuthorizationManager opID=vim-cmd-f7-d204 user=dcui] Enable lockdown mode failed: N3Vim5Fault12UserNotFound9ExceptionE(Fault cause: vim.fault.UserNotFound --> ) --> [context]zKq7AVICAgAAAJXqSgEOaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAfhmT2hvc3RkAAEkoLYBZt21AUEstgE9kbYB7ku6gmGVGAFsaWJ2aW0tdHlwZXMuc28AAUqbUgDMxy0AVAMuAAIRPwM7fQBsaWJwdGhyZWFkLnNvLjAABG3RDmxpYmMuc28uNgA=[/context]
YYYY-MM-DDTHH:MM:SS.792Z info hostd[2099935] [Originator@6876 sub=Vimsvc.AuthorizationManager opID=vim-cmd-f7-d204 user=dcui] Restoring the removed permissions...
YYYY-MM-DDTHH:MM:SS.801Z info hostd[2100829] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-f7-d204 user=dcui] Event 5079 : Permission created for root on root, role is Administrator, propagation is Enabled 2023-08-22T06:20:41.800Z info hostd[2099935] [Originator@6876 sub=AdapterServer opID=vim-cmd-f7-d204 user=dcui] AdapterServer caught exception; <<52e3717a-d8df-9934-3803-8dcc9e04d43f, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 44121'>>, ha-host-access-manager, vim.host.HostAccessManager.changeLockdownMode>, N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError --> ) --> [context]zKq7AVICAgAAAJXqSgEMaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAfhmT2hvc3RkAAGa9lEBdJS2Ae5LuoJhlRgBbGlidmltLXR5cGVzLnNvAAFKm1IAzMctAFQDLgACET8DO30AbGlicHRocmVhZC5zby4wAARt0Q5saWJjLnNvLjYA[/context]
YYYY-MM-DDTHH:MM:SS.804Z info hostd[2099935] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-f7-d204 user=dcui] Task Completed : haTask-ha-host-vim.host.HostAccessManager.changeLockdownMode-123333759 Status error
YYYY-MM-DDTHH:MM:SS.804Z info hostd[2099935] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-f7-d204 user=dcui] Activation finished; <<52e3717a-d8df-9934-3803-8dcc9e04d43f, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 44121'>>, ha-host-access-manager, vim.host.HostAccessManager.changeLockdownMode>
YYYY-MM-DDTHH:MM:SS.804Z verbose hostd[2099935] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-f7-d204 user=dcui] Arg mode: --> "lockdownNormal"
YYYY-MM-DDTHH:MM:SS.805Z info hostd[2099935] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-f7-d204 user=dcui] Throw vmodl.fault.SystemError
YYYY-MM-DDTHH:MM:SS.805Z info hostd[2099935] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-f7-d204 user=dcui] Result: --> (vmodl.fault.SystemError) { --> reason = "Internal error", --> msg = "", --> }

/var/log/vmware/vpxd/vpxd.log

p0x00007fdcb809b980, h:257, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 58674'>>, id: 37696485, state(in/out): 3/1}, session: <52ebcc64-f2a6-3f73-f662-9de8be16f4b6, <TCP '127.0.0.1 : 8085'
>, <TCP '127.0.0.1 : 58674'>>, req: {POST, /sdk}}>, N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
--> )
--> [context]zKq7AVECAQAAAA8jcwEfdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwCMmUoBhakXbGlidm1vbWkuc28AAdYZJwHhYiEB6pAhAV8KIYKz0ksBdnB4ZACCWxhMAYLNI0wBgtUkTAGCl8FLAYKP70sBAdfaGgEnzxqDA05WAWxp
YnZpbS10eXBlcy5zbwCCYbIsAYJtFYICAU11HIKk718CguyggAKCN7GAAoIrxH8CghqbgAIABOw3ABdFOAC7D1EEsI4AbGlicHRocmVhZC5zby4wAAXf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DDTHH:MM:SS.278-08:00 error vpxd[07352] [Originator@6876 sub=Default opID=m6paocl5-6894692-auto-43rza-h5:70717844-4] [VpxLRO] -- ERROR task-3389965 -- 52ebcc64-f2a6-3f73-f662-9de8be
16f4b6(526720a9-7ad7-899b-8f2b-92f3f7a4546c) -- hostAccessManager-47161 -- vim.host.HostAccessManager.changeLockdownMode: :vmodl.fault.SystemError
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Internal error"
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fdc88b5ed58, h:238, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-47161/vpxa>]: changeLockdownMode
--> Received SOAP response fault from [<<io_obj p:0x000000371123b898, h:26, <TCP '127.0.0.1 : 29810'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: changeLockdownMode
--> A general system error occurred: Internal error"
--> }
--> Args:
-->
--> Arg mode:
--> "lockdownNormal"

Resolution

For ESXi 7.0.2:
===============

# Stop hostd
$ /etc/init.d/hostd stop

# Restore the default permission.
$ cp /etc/vmware/hostd/.#authorization.xml /etc/vmware/hostd/authorization.xml

# Start hostd
$ /etc/init.d/hostd start

For ESXi 7.0.3:
===============

# Stop hostd
$ /etc/init.d/hostd stop

# Dump the default host permissions from the config store to a json file
$ /bin/configstorecli config default get -c esx -g authorization -k permissions -outfile /tmp/tmp.json

# Delete all permissions from config store
$ /bin/configstorecli config current delete --all -c esx -g authorization -k permissions

# Update the config store with the previous json file.
$ /bin/configstorecli config current set -c esx -g authorization -k permissions -infile /tmp/tmp.json

# Start hostd
$ /etc/init.d/hostd start

 

Workaround:

#esxcli system permission unset -i '' --group
#esxcli system permission list
#vim-cmd -U dcui vimsvc/auth/lockdown_mode_enter