Missing ESXi service accounts in SDDC Manager after Bringup

Products

VMware Cloud Foundation

Issue/Introduction

This article provides the steps to recreate the missing ESXi service accounts in SDDC Manager.

Symptoms:

Bringup is run/retried multiple times.
The last run did not populate ESXi service accounts in SDDC Manager. However, the accounts were created on the hosts as part of the first run.

Environment

VMware Cloud Foundation 4.1
Vmware Cloud Foundation 5.1
VMware Cloud Foundation 4.2

Cause

The first run of Bringup failed without persisting the ESXi service accounts, that were created on the ESXi hosts.

Resolution

Scenarios where the ESXi Service Account is missing after bringup is resolved in VCF 4.3.

For other scenarios, please review the workaround section for steps to unblock.

Workaround:
To work around this issue:
Note: For VCF 5.1, please use the full path /usr/pgsql/13/bin/psql to invoke the psql command.

Scenario 1: If the issue is observed after Bringup is completed

SSH on SDDC Manager.

Run the below command to fetch the esxi-id

psql -h localhost -d platform -U postgres -c "select id from host where id not in (select entityid from credential where targettype='ESXI' and serviceid is not null);"

On SDDC Manager, create /tmp/in.json with the following content:

The list should contain an entry for each missing ESXi service accounts. For example: below list contains details for two accounts.

[
  {
    "username":"svc-vcf-<esxi-shortname>",
    "roleName":"Admin",
    "resourceType":"ESXI",
    "resourceId":"<esxi-id>"
  },
  {
    "username":"svc-vcf-<esxi-shortname>",
    "roleName":"Admin",
    "resourceType":"ESXI",
    "resourceId":"<esxi-id>"
  }
]

Note: For SDDC manager 5.1, the .json file should contain the following:
{
"entityType" : "ESXI",
"targetType" : "ESXI",
"serviceId" : "11847e00-c881-4178-ad1a-3b0071c7d60a",
"username" : "svc-vcf-s623918ch3cf01",
"serviceType" : "SDDC_MANAGER",
"entityId" : "e1e4639e-f7f6-46b3-847f-6c5c1594741a",
"credentialType" : "SSH",
"secret": "<password>",
"id": "4f9e6a41-c1ed-4cd9-bc84-b93d4011e0f1"

where:
"serviceId" <----- SDDC Manager ID
"username" <----- the username of the account created on the host
"entityId" <----- the ID of the host
"secret": <---- the password set on the host
"id": <--- randomly generate it (or change a few symbols from the example)

SDDC Manager ID:
log in to SDDC Manager VM, and run:
psql -h localhost -U postgres -d platform -c "select id from sddc_manager_controller ;"

Mapping between hosts and their IDs:
psql -h localhost -U postgres -d platform -c "select id, hostname from host;"

For each missing ESXi service account in SDDC Manager, remove the local ESXi user:

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@<esxi-FQDN> "esxcli system account remove -i svc-vcf-<esxi-shortname>"

Generate API token:

curl -X POST -H "Content-Type: application/json" -d '{"username": "<SSO-username>","password": "<SSO-password>"}' --insecure https://<sddc-manager-address>/v1/tokens | json_pp

Example:
Token Generation API Response:
{ "accessToken": "###########################", "refreshToken": { "id": "########-####-####-####-############" } }

We would replace the <API-TOKEN> in the following steps with just the access token generated above:

Create the ESXi service account/s

curl -X POST https://<sddc-manager-address>/v1/service-accounts -d @/tmp/in.json --insecure -H "Content-Type: application/json" -H "Authorization: Bearer <API-TOKEN>"

Response:
{"id":"<task-id>","name":"Creating Service Accounts","status":"IN_PROGRESS" ...}

Wait for the workflow from 5. to complete by tracking its status:

curl https://<sddc-manager-address>/v1/tasks/<task-id> --insecure -H "Content-Type: application/json" -H "Authorization: Bearer <API-TOKEN>"

Get the {service-id} from the database:

psql -h localhost -d platform -U postgres -c "select serviceId from credential where targettype='ESXI';"

Update serviceid and servicetype in the database:

psql -h localhost -d platform -U postgres -c "update credential SET serviceid='<service-id>', servicetype='SDDC_MANAGER' where username='svc-vcf-<esxi-shortname>';"

Scenario 2: If you have noticed that the ESXi hosts have leftover service accounts before starting Bringup

Note: You can check on the ESXI hosts esxcli system account list.

For each host with a leftover service account, execute step 3.