Host Commissioning Fails Due to Stale ESXi Service Accounts - SDDC Manager
search cancel

Host Commissioning Fails Due to Stale ESXi Service Accounts - SDDC Manager

book

Article ID: 451550

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

  • Commissioning fails at "Create New Local ESX Host User" stage. Configuration status displays as "Error".
  • Commissioning the hosts which had already gone through multiple commission and decommission cycles on the SDDC Manager fails at the "Create New Local ESX Host User" stage.

    All hosts are skipped, failing the workflow



  • Configuration status displays as "Error". 



  • During commissioning, SDDC Manager attempts to create a dedicated local service account (svc-####) on each target ESXi host.
  • Incomplete decommissioning from previous cycles leaves stale service accounts on the host. When SDDC Manager attempts to validate access using newly generated credentials, authentication fails due to a credential mismatch with the existing stale user.
  • SDDC Manager: /var/log/vmware/vcf/operationsmanager/operationsmanager.log file confirms that the SDDC Manager is attempting to create a service account on the ESXi host during the commissioning workflow. However, it detected that the user already exists and the user name and the password for the existing svc user is incorrect. 

    YYYY-MM-DDTHH:MM:SS.MS+HH:MM INFO  [vcf_om,################################,b4b1] [c.v.e.s.v.contract.CreateEsxiUser,pool-2-thread-19] Creating local user <USER> in ESXi host ###-####-####-##.####.#####
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM DEBUG [vcf_om,################################,b4b1] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-2-thread-19] Existing ESXi users on ###-####-####-##.####.##### are: [root, dcui, vpxuser, svc-###-###-###-###-###]
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM WARN  [vcf_om,################################,b4b1] [c.v.e.s.v.contract.CreateEsxiUser,pool-2-thread-19] User <USER> already exists
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM INFO  [vcf_om,################################,b4b1] [c.v.e.s.v.contract.CreateEsxiUser,pool-2-thread-19] Checking if user <USER> is operational with given credentials
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM ERROR [vcf_om,################################,b4b1] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-2-thread-19] Unable to connect to svc-###-###-###-###-###@###-####-####-##.####.#####
    com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user <USER> or password.
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM ERROR [vcf_om,################################,4814] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-2-thread-8] Cannot complete login due to incorrect credentials: ###-####-####-##.####.#####, svc-###-###-###-###-###.
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM ERROR [vcf_om,################################,4814] [c.v.e.s.v.contract.CreateEsxiUser,pool-2-thread-8] Unable to login to ###-####-####-##.####.##### as svc-###-###-###-###-### user: <USER> complete login due to an incorrect user <USER> or password.
    com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user <USER> or password.
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM ERROR [vcf_om,################################,b4b1] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-2-thread-19] Failed to connect to <URL> as svc-###-###-###-###-###
    com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user <USER> or password.Log into the affected ESXi host via SSH using root credentials.
    Remove the stale service account: esxcli system account remove -i svc-###-###-###-###-###
    Retrigger the host commissioning workflow in SDDC Manager.

  • The /var/run/log/hostd.log file confirms that the ESX affected host confirms that host PAM authentication rejected the login attempt.

    YYYY-MM-DDTHH:MM:SS.MSZ -ERROR Hostd ####### [esx@#### sub="Default" opID="c8d96###" rhost="##.#.##.#" sid="########"] [module:pam_lsass]pam_do_authenticate: error [login:#][error code:2]
    YYYY-MM-DDTHH:MM:SS.MSZ -ERROR Hostd ####### [esx@#### sub="Default" opID="c8d96###" rhost="##.#.##.#" sid="########"] [module:pam_lsass]pam_sm_authenticate: failed [error code:2]
    YYYY-MM-DDTHH:MM:SS.MSZ -WARNING Hostd ####### [esx@#### sub="Vimsvc.HaSessionManager" opID="c8d96###" sid="########"] Rejected password for user <USER> from ##.#.##.# - session=########-####-####-####-############
    YYYY-MM-DDTHH:MM:SS.MSZ -INFO Hostd ####### [esx@#### sub="Vimsvc.ha-eventmgr" opID="c8d96###" sid="########"] Event 4119 : Cannot login svc-###-###-###-###-###@##.#.##.#Log into the affected ESXi host via SSH using root credentials.
    Remove the stale service account: esxcli system account remove -i svc-###-###-###-###-###
    Retrigger the host commissioning workflow in SDDC Manager.

  • Running the command "esxcli system account list" on the affected host confirms an existing stale svc account.

    esxcli system account list

    User ID                   Description                                  Shell access  Last password change
    ------------------------  -------------------------------------------  ------------  --------------------
    svc-###-###-###-###-###   ESXi User                                            true  2026-07-26

Environment

  • VCF 9.x

Cause

Incomplete decommissioning leaves stale service accounts (e.g., svc-vcf-pdc-XXXXXXX) on the host, causing credential mismatches.

Resolution

  1. Log into the affected ESXi host via SSH using root credentials.
  2. Remove the stale service account:

    esxcli system account remove -i svc-###-###-###-###-###

  3. Retrigger the host commissioning workflow in SDDC Manager.

Additional Information

Steps to Decommission ESX Hosts

Powered by Wolken Software