ESXi upgrade precheck fails with an error pertaining to a missing service account.

Products

• VMware Cloud Foundation

Issue/Introduction

• The precheck fails with the following UI error:

Message: Service account to connect to the resource : <ESXi_Hostname> couldn't be found.

Remediation Message: Please ensure that a service account exists to be able to be used to connect to the ESXi host.
 

• On SDDC Manager, /var/log/vmware/vcf/operationsmanager/operationsmanager.log contains the following logging (if a failed Password Management operation involving this host was made):

2023-04-26T12:21:28.004+0000 DEBUG [vcf_om,3d757285b7d57ad0,82f5] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-20] About to do testBeforeRemediation step..
2023-04-26T12:21:28.004+0000 DEBUG [vcf_om,3d757285b7d57ad0,82f5] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-20] Remediation required: Performing test before remediation for username: root
2023-04-26T12:21:28.004+0000 ERROR [vcf_om,3d757285b7d57ad0,82f5] [c.v.v.p.update.changers.EsxiChanger,om-exec-20] No service account to connect to the host : <ESXi_Hostname> was found !
2023-04-26T12:21:28.004+0000 ERROR [vcf_om,3d757285b7d57ad0,82f5] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-20] Service account to connect to the resource : <ESXi_Hostname> be found.
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Service account to connect to the resource : <ESXi_Hostname> be found.

• /var/log/vmware/vcf/lcm/lcm-debug.log contains the following for the failed precheck:

2023-04-26T12:47:59.024+0000 ERROR [vcf_lcm,2ba9d78398dd41f4,f6cf,precheckId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx,resourceType=ESX,resourceId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx] [c.v.e.s.l.a.i.i.CredentialProviderEvo,Prec
heck-3] Credential retrieval failed for targetType ESXI and serviceType SDDC_MANAGER and entityId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : "{"errorCode":"CREDENTIAL_REST_INTERNAL_SERVER_ERROR","arguments":[],"message":"Credential Internal Server Error","causes":[{"type":"java.la
ng.NullPointerException"}],"referenceToken":"XXXXX"}"
 

• The service account in question is present on the host.
• The service account details are partially present in the credentials table in the Postgres database on SDDC Manager.
• The serviceid and servicetype columns in the credential table are blank for the affected host(s) but not for functioning hosts.

Environment

VMware Cloud Foundation 4.x

Cause

The most likely cause of this is the incorrect implementation of the steps found in the following KB (steps 7 & 8 from Scenario 1 were not completed):

Missing ESXi service accounts in SDDC Manager after Bringup
https://broadcomcms-software-agent.wolkenservicedesk.com/wolken/esd/knowledge-base-view/view-kb-article?articleNumber=318749

Resolution

Although various factors may be involved, the most likely cause of this is user error, as described in the Cause section of this article.

 

Note: Ensure to take a snapshot of the SDDC Manger before making any changes.

Workaround:
To resolve this, the missing values must be added to the credential table:

1. Access the Platform database in Postgres with the following command:

psql -h localhost -U postgres -d platform

2. Collect the serviceid from the SDDC Manager table:

select id from sddc_manager_controller;

3. Update the affected values for the affected service account (replace values with those collected in step 2):

update credential set serviceid = 'id_from_step2' where username = 'esxi_service_account';
update credential set servicetype = 'SDDC_Manager' where username = 'esxi_service_account';

4. Restart the LCM service with the following command:

systemctl restart lcm

5. Retry the failed precheck in the SDDC Manager UI.