VMware HCX 9.1 VcenterSecretRotationJob Fails With Directory Error 87

Products

• VMware HCX

Issue/Introduction

In VMware HCX 9.1, the periodic VcenterSecretRotationJob may fail every cycle when attempting to rotate the connection secret via the vCenter Server svcaccountmgmt API. While standard operations such as inventory sync and topology polling remain healthy, the logs show continuous rotation failures.

• The HCX app.log shows a negative value for secret expiration (e.g., Secret rotation policy is expiring in -495813: hours!).
• The rotation job fails with the following error:

  {"statusCode":500,"reason":"Internal Server Error", "result":{"error_type":"ERROR","messages":[{"args":["Internal Server Error, VMware directory error[87]"], "id":"com.vmware.vcenter.svcaccountmgmt.error"}]}}

• The job retries and logs this error at frequent intervals (every 30–90 seconds).

Environment

VCF 9.1.0.x

VMware HCX 9.1.0.x

Cause

Automatic password rotation for service accounts was introduced in HCX 9.1. The svcaccountmgmt API in vCenter is designed to operate on specific Service Account objects. It does not support secret rotation for named SSO users like [email protected].

When HCX attempts to rotate the password for a named SSO user, the vCenter directory (vmdir) returns error[87], which corresponds to a Win32 ERROR_INVALID_PARAMETER. This occurs because the account name is not a valid target for the service account management API. The negative expiration timestamp in the logs is a side effect of the API returning a null/epoch value for human-facing accounts. 

Resolution

This issue is resolved in VMware HCX 9.1.1. In this version, the password rotation logic is updated to skip rotation if the vCenter account is a named SSO user or if the password policy is set to "Never Expire."

Workaround: If an upgrade is not immediately possible, you can reduce the frequency of the failure logs by increasing the rotation job interval:

1. Log in to the HCX Manager via SSH as the admin user.
2. Locate the ZQL configuration file:

   /opt/vmware/deploy/config/SecretRotateService.zql

3. Edit the file and update the value for vcenterSecretRotationJobIntervalInHours from 24 to 168 (this changes the check to once every 7 days).
4. Ensure no migration or configuration tasks are currently running.
5. Restart the HCX App Engine service to apply the changes:

   sudo systemctl restart app-engine

Additional Information

• Service-to-service accounts managed by VCF typically use the svc-* prefix.
• Continuous rotation failures of this type are generally cosmetic and do not result in account lockouts for the SSO user, as the failure occurs at the API validation layer before an authentication attempt is made.