Unable to enable lockdown mode in ESXI

Products

VMware vCenter Server VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0 VMware vSphere ESXi

Issue/Introduction

Unable to enable lockdown mode from vCenter, ESXI DCUI or CLI
Following error received:

(vmodl.fault.SystemError) {
   faultCause = (vmodl.MethodFault) null,
   faultMessage = <unset>,
   reason = "Internal error"
   msg = "Received SOAP response fault from [<<io_obj p:0x000000ccce14fa68, h:5, <TCP '127.0.0.1 : 25061'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: changeLockdownMode
A general system error occurred: Internal error"
}

Environment

VMware vCenter Server 7.x
VMware vCenter server 8.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

If ESXI was joined to domain and later removed, Domain users permission might exists on ESXI host
Check the permission using the below commands

#esxcli system permission list

#/bin/configstorecli config default get -c esx -g authorization -k permissions -outfile /tmp/tmp.json

Check for below entries in /var/run/log/hostd.log of ESXI :

:70154847-3b-7c-a60d user=vpxuser:VSPHERE.LOCAL\Administrator] Group lookup failed for 'XXXXXX\esx^admins'
hostd.2:2024-11-07T06:06:25.447Z warning hostd[2105747] [Originator@6876 sub=Vimsvc opID=m2g2znp2-1898263-auto-14opl-h5:701548
47-3b-7c-a60d user=vpxuser:VSPHERE.LOCAL\Administrator] [ACL] Could not resolve group XXXXXX\esx^admins
hostd.2:2024-11-07T06:06:25.448Z info hostd[2104468] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=m2g2znp2-1898263-auto-14opl-
h5:70154847-3b-7c-a60d user=vpxuser:VSPHERE.LOCAL\Administrator] Event 532 : Permission rule removed for da-user on root
hostd.2:2024-11-07T06:06:25.452Z info hostd[2105726] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=m2g2znp2-1898263-auto-14opl-
h5:70154847-3b-7c-a60d user=vpxuser:VSPHERE.LOCAL\Administrator] Event 533 : Permission rule removed for cloudadmin on root
hostd.2:2024-11-07T06:06:25.452Z warning hostd[2105747] [Originator@6876 sub=UserDirectory opID=m2g2znp2-1898263-auto-14opl-h5
:70154847-3b-7c-a60d user=vpxuser:VSPHERE.LOCAL\Administrator] Group lookup failed for 'XXXXXX\esx^admins'
hostd.2:2024-11-07T06:06:25.453Z error hostd[2105747] [Originator@6876 sub=Vimsvc.AuthorizationManager opID=m2g2znp2-1898263-a
uto-14opl-h5:70154847-3b-7c-a60d user=vpxuser:VSPHERE.LOCAL\Administrator] Cannot remove ACE: N7Vmacore9Authorize27AuthUserUnr
esolvedExceptionE(Group XXXXXX\esx^admins)
hostd.2:2024-11-07T06:06:25.458Z error hostd[2105747] [Originator@6876 sub=Vimsvc.AuthorizationManager opID=m2g2znp2-1898263-a
uto-14opl-h5:70154847-3b-7c-a60d user=vpxuser:VSPHERE.LOCAL\Administrator] Enable lockdown mode failed: N3Vim5Fault12UserNotFo
und9ExceptionE(Fault cause: vim.fault.UserNotFound

esx^admins is not removeable with below commands ,As ESXI is no part of domain

#esxcli system permission unset -i 'XXXXXXX\esx^admins' --group

Resolution

Follow the below steps,

Stop hostd

/etc/init.d/hostd stop

Dump the default host permissions from the config store to a json file

/bin/configstorecli config default get -c esx -g authorization -k permissions -outfile /tmp/tmp.json

Delete all permissions from config store

/bin/configstorecli config current delete --all -c esx -g authorization -k permissions

Update the config store with the previous json file.

/bin/configstorecli config current set -c esx -g authorization -k permissions -infile /tmp/tmp.json

Start hostd

/etc/init.d/hostd start

Enable Lockdown Mode from vCenter UI or from DCUI (Direct Console User Interface)