Steps to recover expired Service Accounts in VMware Cloud Foundation

Products

• VMware Cloud Foundation

Issue/Introduction

This article provides information on how to recover the service accounts when they expire.

Symptoms:
When service accounts are expired, certain VCF workflows like Domain creation, expansion, would fail and Consumers of SSO accounts like NSXT and VRSLCM would fail to communicate with vCenter Server.

Environment

VMware Cloud Foundation 4.1
VMware Cloud Foundation 4.2
VMware Cloud Foundation 4.4.x

Cause

For VCF 4.1 and 4.2 there is no Auto-rotate password feature for every 'X' days for service accounts. This could lead to expiry of accounts.

Service accounts on vCenter Server expires for every 90 days.

operationsmanager.log
root account remediate/rotation fails after resetting root password on ESXi.

2023-06-29T15:29:52.460-0400 INFO  [vcf_om,6acb08a361a64915,4adb] [c.v.v.p.r.CancelPasswordTransactionHandler,http-nio-127.0.0.1-7300-exec-9] Cancelling password rotate for entity..{"transactionId":453,"entityName":"XXXXXXX.XXXXXXX.XXXXXXX","entityId":"1b1f75dc-93ce-4bf1-9b1f-c6b51ac0128","oldPassword":"*****","newPassword":"*****","entityType":"ESXI","credentialType":"*****","transactionStatus":"FAILED","transactionTime":"Jun 29, 2023, 3:29:36 PM","updateStage":"TEST_BEFORE_REMEDIATE","workflowId":"f31e2775-92e3-46c3-9d4e-abe8a27f5090","username":"root","diagnosticMessage":"{\"errorCode\":\"PASSWORD_MANAGER_VALIDATE_ESXI_CREDENTIALS_FAILED\",\"arguments\":[\"*****\"],\"errorMessage\":\"(vim.fault.PasswordExpired) {\\n   faultCause = null,\\n   faultMessage = null\\n}\",\"updateStage\":\"*****\",\"referenceToken\":\"BQDJV6\"}"}

Resolution

To resolve this issue:

To recover Service account expired on vCenter SSO :

1. Unlock and reset SSO account in VCenter. Please refer vCenter Admin Guide
2. Open SDDC Manager→ Security→ Password Management UI
3. Perform 'Remediate' operation providing the new password used in step 1.

Note:

• Password management will tests/sync internal data store and applies new auto generated password for security reasons.
• When Remediate operation completed, then VCF Workflows should go through and connection status to vCenter from Consumers should be green.

Service accounts on ESXI follow same policy as 'root' account, so there is no expiry days set and hence never expires unless you manually set.

For service account of ESXi expires, when expiry days set manually:

1. Login to ESXI host using root account.
2. Reset password for service account svc-xxx using passwd command:

passwd svc-xxxx 

3. Reset failed logins to 0 using the command:

pam_tally2 command ex pam_tally2 --user=svc-xxxx --reset4. Open SDDC Manager→ Security→ Password Management UI

4. Perform 'Remediate' operation providing the new password used in step 2.

Note: Password management will   tests/sync internal data store and applies new auto generated password for security reasons.

Additional Information

Service accounts are the accounts used by VCF components for inter communication. These are not user accounts and hence are not meant for users to consume these accounts.

For VCF 4.1, VCF has 2 sets of service accounts.

Service accounts created on ESXI hosts
Service accounts created on vCenter as SSO accounts, where NSXT and VRSLCM are consumers.
Username of these service accounts would start from svc-:

Example:

• For ESXI host, svc-vcf-esxi-1
• For SSO vCenter [email protected]