Unable to commission host to SDDC Manager, task is getting failed on subtask 'Add Service Account to ESXi Lockdown Mode Exception Users List'

Products

• VMware SDDC Manager

Issue/Introduction

Below log entries are observed,

 

SDDC Manager
operationsmanager.log
yyyy-mm-ddThh:mm:ss.672+0000 INFO [vcf_om, ldcae826b821dbc6, 5fee] [c.v.e.s.v.c.AddServiceAccountToLockdownModeExceptionUsersList, pool-2-thread-12] Adding user svc-vcf-esxi01 to ESXi host esxi01.che.dc.tbintra.net lockdown mode exception users.
yyyy-mm-ddThh:mm:ss.716+0000 ERROR [vcf_om, ldcae826b821dbc6, 5fee] [c.v.e.s.v.c.AddServiceAccountToLockdownModeExceptionUsersList, pool-2-thread-12] All hosts are in skip failed hosts list, failing workflow
yyyy-mm-ddThh:mm:ss.717+0000 ERROR [vcf_om, ldcae826b821dbc6, 5fee] [c.v.e.s.o.model. error.ErrorFactory, pool-2-thread-12] [V80HI2] ALL_HOSTS_SKIPPED_FAILED All hosts are skipped, failing the workflow.
com. vmware. evo. sddc. orchestrator. exceptions. OrchTaskException: All hosts are skipped, failing the workflow.
at com. vmware. evo.sddc. vsphere. contract.AddServiceAccountToLockdownModeExceptionUsersList. skipFailedHosts (AddServiceAccountToLockdownModeExceptionUsersList. java: 202)

ESXi
hostd.log
yyyy-mm-ddThh:mm:ss.695Z In (166) Hostd[2099914]: [Originator@6876 sub=Vimsvc. TaskManager opID=5867952e sid=52c539f7 user=root] Task Created : haTask-ha-host-vim. host . HostAccessManager . updateLockdownExceptions-1802723723
yyyy-mm-ddThh:mm:ss.696Z Wa (164) Hostd[2099932]: [Originator@6876 sub=UserDirectory opID=5867952e sid=52c539f7 user=root] User lookup failed for 'svc-vcf-esxi01-stale'
yyyy-mm-ddThh:mm:ss.696Z Er (163) Hostd[2099932]: [Originator@6876 sub=Vimsvc.AuthorizationManager opID=5867952e sid=52c539f7 user=root] User not found: N7Vmacore9Authorize25AuthUserNotFoundExceptionE (User svc-vcf-esxi01-stale)
..
..
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099932]: [Originator@6876 sub=Vimsvc. TaskManager opID=5867952e sid=52c539f7 user=root] Task Completed : haTask-ha-host-vim. host. HostAccessManager. updateLockdownExceptions-1802723723 Status error
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099932]: [Originator@6876 sub=Solo. Vmomi opID=5867952e sid=52c539f7 user=root] Activation finished; << 52c539f7-c4fl-eecf-794c-690ee485b758, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 62214'>>, ha--access-manager, vim. host. HostAccessManager . updateLockdownExceptions, <vim. version. v7_0, internal, 7.0.0.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE : 0x000000fe0ae13078]>
yyyy-mm-ddThh:mm:ss.709Z Db (167) Hostd[2099932]: [Originator@6876 sub=Solo. Vmomi opID=5867952e sid=52c539f7 user=root] Arg users:
yyyy-mm-ddThh:mm:ss.709Z Db (167) Hostd[2099874]: -- > (string) [
yyyy-mm-ddThh:mm:ss.709Z Db (167) Hostd[2099874]: -- >   "svc-vcf-esxi01-stale",
yyyy-mm-ddThh:mm:ss.709Z Db (167) Hostd[2099874]: -- >   "svc-vcf-esxi01"
yyyy-mm-ddThh:mm:ss.709Z Db (167) Hostd[2099874]: -- > ]
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099932]: [Originator@6876 sub=Solo. Vmomi opID=5867952e sid=52c539f7 user=root] Throw vim. fault. UserNotFound
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099932]:
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099874]: -- > (vim. fault. UserNotFound) {
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099874]: -- >  principal = "svc-vcf-esxi01-stale",
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099874]: -- >  unresolved = false,
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099874]: -- >  msg = "",
yyyy-mm-ddThh:mm:ss.709Z In (166) Hostd[2099874]:

Environment

SDDC 5.x

Cause

Their is a duplicate/stale ESXi service account present in the lockdown mode exception users list. 

Confirmed this through ESXi host UI with below click path,
ESXi host UI >> Manage >> Security & Users >> Lockdown Mode.

SDDC has created the svc account as 'svc-vcf-esxi01' based on the ESXi FQDN.

However 'svc-vcf-esxi01-stale' is old stale entry of user already added in the lockdown mode exception users list.

Resolution

Remove the stale/duplicate entry of the svc-vcf-sarsafc31010106 user from lockdown mode exception users list from the host UI Client.

 

ESXi host UI >> Manage >> Security & Users >> Lockdown Mode.

Select the existing stale user account, click Remove user Exception.

Logout and re login to SDDC UI.

Restart the task of host commissioning.