UpdateDatastoreSource API fails for Cross vCenter service account password reset

Products

• VMware vSAN

Issue/Introduction

• In a Cross-vCenter environment, attempting to reset the service account password for a server vCenter by invoking the UpdateDatastoreSource API on the client vCenter will fail.
• This issue occurs when a client cluster mounts a remote vSAN datastore managed by a separate vCenter instance, and that remote datastore's owner vCenter is running version 9.1.0.0 EP2.

• The client vCenter logs located at /var/log/vmware/vsan-health/vmware-vsan-health-service.* contain the following error message:

  xxxx-xx-xxTxx:xx:xx.xxxZ ERROR vsan-mgmt 32208 [vc@4413 tid="58885" func="ServiceAccountHelper::_convertFault" opID="e4397372"] Receive fault about operating service account
Traceback (most recent call last):
  File "bora/vsan/xvc/lib/ServiceAccountHelper.py", line 62, in _convertFault
  File "bora/vsan/xvc/lib/ServiceAccountHelper.py", line 240, in ResetServiceAccountPassword
  File "/usr/lib/vmidentity/svcaccountmgmt/python-bindings/com/vmware/vcenter/svcaccountmgmt_client.py", line 407, in reset File "/usr/lib/vmware-vpx/vsan-health/vapi/vapi_runtime-2.100.0.egg/vmware/vapi/bindings/stub.py", line 408, in _invoke
  File "/usr/lib/vmware-vpx/vsan-health/vapi/vapi_runtime-2.100.0.egg/vmware/vapi/bindings/stub.py", line 357, in native_invoke
com.vmware.vapi.std.errors_client.NotFound: {messages : [LocalizableMessage(id='com.vmware.vcenter.svcaccountmgmt.notfound', default_message='Service account not found for given name (Service account not found:

Note that all of the following conditions in the "Environment" section and the symptoms above including the error in the vmware-vsan-health-service logs must be met to confirm this issue

Environment

• VMware vSAN 9.1.0 EP2
• The environment is configured for Cross-vCenter remote datastore mounting.
• The vCenter managing the remote datastore is running version 9.1.0.0 EP2

Cause

• This issue occurs because the password reset API enforces strict validation on the service account name format in 9.1.0.0 EP2.
• The specific account name format passed by the UpdateDatastoreSource API does not comply with this validation rule, causing the account to not be recognized.

Resolution

The issue is resolved in vCenter Server 9.1.1

Workaround:

To workaround this issue, the service account password can be reset manually by executing a script located at the client vCenter Server.

1. Log in to the client vCenter Server (the vCenter hosting the cluster that mounts the remote datastore) via SSH as root.
2. Navigate to the script directory: 
   /usr/lib/vmware-vpx/vsan-health/bin
3. To reset the password, run: 
   ./ServiceAccountCtl.sh -r -h '[server vcenter IP]' -u '[email protected]' -p '[server vcenter password]'
4. To verify whether resetting the password worked well, run: 
   ./ServiceAccountCtl.sh -v -h '[server vcenter IP]'
5. To reset and verify the password at the same time, run: 
   ./ServiceAccountCtl.sh -r -v -h '[server vcenter IP]' -u '[email protected]' -p '[server vcenter password]'

Note: Replace [email protected] with your local SSO admin user if the local SSO domain is different.