Missing SDDC Manager Service account for an ESXi host - VMware Cloud Foundation

Products

• VMware Cloud Foundation
• VMware vSphere ESXi

Issue/Introduction

The purpose of this KB is to provide the steps to re-create the missing service account on an ESXi host, if it already exists on the SDDC Manager.

Note: If the service account does not exist on the SDDC Manager as well, please use the following KB instead: Missing ESXi service accounts in SDDC Manager after Bringup

• Missing the SDDC Manager created service account on the ESXi hosts.
• Unable to perform any administrative tasks against the ESXi host via the SDDC Manager (e.g. Upgrade Prechecks show errors for the host) 
• On the SDDC Manager, these service accounts appear to be disconnected. 
• Config drift upgrade will fail at vSAN LSOM update stage on certain hosts.
  • LCM logs will report:
    Failed to update vSAN LSOM settings on the host esxi_shortname.example.com
Failed to connect to esxi_shortname.example.com

Environment

• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x
•  
  VMware Cloud Foundation 9.x

Cause

If the host has been rebuilt/reconfigured and then re-added to the vCenter without going through the decommission / commission process from the SDDC Manager, the service account may be missing on the ESXi host.

Resolution

Check if the service account is missing. Run the following command in a SSH session on the affected ESXi host: 

• esxcli system account list

Sample output:

        [root@esxi_shortname:~] esxcli system account list
    User ID         Description                                Shell access
    --------------  -----------------------------------------  ------------
    root            Administrator                                      true
    dcui            DCUI User                                          true
    vpxuser         VMware Workstation administration account          true
    cloudadmin      ESXi User                                          true
    lldpVim-user    ESXi User                                          true
    mux_user        ESXi User                                          true
    da-user         ESXi User                                          true
    nsx-user        ESXi User                                          true

Since the svc-vcf-esxi_shortname account is missing, manually re-create the service account on the ESXi host.

NOTE: Confirm if the service account is missing from SDDC.  If it is present in the SDDC Manager, use its password credentials in the Step 2 below when recreating the service account:

Retrieve the service accounts credentials from SDDC Manager

METHOD 1: Create the missing service account from the ESXi Host Client UI by logging into the ESXi host with root or administrator level credentials

1. 1. 1. In the host client on the first ESXI host, navigate to Manage > Security & Users > Users.
      2. Click Add User.
      3. Provide username (svc-vcf-esxi_shortname) and password (See above note and use the password that SDDC Manager has for this account).
      4. Add the administrator administrator role to the service account.
      5. Confirm you can log into the ESXI Host Client UI using the svc-vcf service account user and password you just set

METHOD 2: Create the missing service account using esxcli commands by logging into the ESXi host via SSH with root credentials

• • NOTE: The account will also be missing from the /etc/passwd and /etc/shadow files

1. 1. 1. Get an SSH session to the affected ESXi host(s)
         
         
      2. Create the service account:
         esxcli system account add --id svc-vcf-esxi_shortname --password '<password>' --password-confirmation '<password>'
         
         
      3. Add the required permission:
         esxcli system permission set --id svc-vcf-<esxi_shortname> --role Admin

      4. Permission for svc-vcf-esxi01 should look like this:
         [root@esxi_shortname:~] esxcli system permission list
Principal       Is Group  Role      Role Description
--------------  --------  --------  ----------------
svc-vcf-esxi_shortname     false  Admin     Full access rights
         
         
      5. Confirm you can log into the ESXI Host Client UI using the svc-vcf service account user and password you just set
         
         

Confirm you can now Remediate the svc-vcf service account password which will sync the account and make it show a status of ACTIVE

• Login to the SDDC Manager UI, and go to the Password Manager
• Locate the svc-vcf service account for the host in question and REMEDIATE the password, using the password supplied while creating the account above.
  • NOTE: Remediating the service account using the password that is already set in the SDDC Manager will produce a warning banner, this can be ignored.
    
    
• (Optional) Rotate the password for the svc-vcf account to confirm that SDDC Manager is able to interact with the host.

VCF service account management issue is fixed in VCF 5.2.4 and also will be part of future 9.1.x release.

To download VMware Cloud Foundation (VCF) version 5.2.4, log into the Broadcom Support Portal

 

*****  IMPORTANT *****

Post-Recreation Verification and Root Account Sync After re-creating the missing service account and remediating its password in the SDDC Manager Password Manager, it is essential to ensure the root account is also synchronized:

1. Log in to the SDDC Manager UI.
2. Navigate to Administration > Password Manager.
3. Verify the status of the root account for the affected host(s).
4. If the root account status is "Disconnected" or showing errors, initiate a remediation for the root account credentials to ensure full connectivity.
5. Retry the DNS/NTP configuration update task.

 

 

 

 

Additional Information

Synchronize the SDDC Manager Service Accounts on the ESXi Host with a PowerShell Script.