Service account delete fails with "Failed to delete permissions" when the serviceaccountmgmt password expires - vCenter Server Appliance 9.1.x

Products

• VMware Cloud Foundation
• VMware vCenter Server

Issue/Introduction

Operations that involve deleting or rotating service accounts fail. This includes:

• NSX compute manager edit fails
• VCFA "Refresh VCF Instances" fails.
• SDDC Manager password rotation fails.
• In '/var/log/vmware/sso/svcaccountmgmt.log' on vCenter, you see errors indicating 'Failed to delete permissions' with an underlying authentication failure. Depending on the caller, the API error may surface as 'Unauthorized' or 'InternalServerError'.
• In '/var/log/vmware/vmdird/vmdird.log' on vCenter server, you observe 'LoginBlocked 9239' errors for the 'serviceaccountmgmt-<machine-id>' user.
• In the Secure Token Signing (STS) logs, you see 'User password expired' for the 'serviceaccountmgmt-<machine-id>'  user.

Environment

vSphere Cloud Foundation 9.1.x

Cause

This issue occurs when the internal 'serviceaccountmgmt' service user's password has expired. When the system attempts to delete a service account, it tries to log in to the 'vpxd' service using this stored password to remove the associated permissions. Login fails due to expired password, causing the delete operation to fail.

Resolution

This issue will be resolved in a future release of vCenter Server.

To work around this issue, it is possible to force the recreation of the 'serviceaccountmgmt' account.

This process requires restarting the Secure Token Signing (STS) service which temporarily interrupts authentication.  It is recommended to perform this procedure in an appropriate maintenance window:

1. Check whether the serviceaccountmgmt NDU marker exists:
   1. ls -l /var/cache/svcaccounts/ndu_marker_files/serviceaccountmgmt
   2. If the file or directory does not exist, proceed to the next step. Do not create it. A marker tells startup to skip service-account setup and can prevent recovery. If it exists, stop this procedure and contact support to verify the upgrade state before changing the marker.
2. With the marker absent, restart STS:
   1. service-control --restart vmware-stsd
3. Confirm STS is running:
   1. service-control --status vmware-stsd
4. Check for a new account-recreation entry after the restart:
   1. grep VmDirCreateSvcAccount /var/log/vmware/vmdird/vmdird.log | grep serviceaccountmgmt
5. Confirm the timestamp belongs to this restart. An older entry does not prove recovery.
6. Retry the operation that failed and confirm it completes successfully.

**Note:** The recreated account will eventually expire again under the same password policy. This workaround may need to be repeated if the issue recurs before the permanent fix is applied.