VM operations hang due to expired VSM service account password

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
On vCenter Server 8.0U2, when attempting to perform VM operations, the operation hangs and completes after an unknown amount of time or fails to complete.

Examples of affected VM operations:
- Powering on
- Rebooting
- Cloning
- Deployment of VMs via OVA/OVF

In some cases an error message is shown saying

Failed to clone state for the entity '<VM_Template_Name>' on extension vService Manager

for example:

In /var/log/vmware/vsm/vsm.log, you may see:

2023-11-02T23:11:57.450-04:00 INFO [Thread-4] ServiceUtil.java 137 - Acquiring SAML token for user vmware-vsm-########-####-####-####-########[email protected]
2023-11-02T23:11:57.787-04:00 ERROR [Thread-4] SoapBindingImpl.java 185 - SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-vsm-########-####-####-####-########acba, Domain: vsphere.local} Please see the server log to find more detail regarding exact cause of the failure.
..
com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983) [libwstclient.jar:?]
      at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) [libwstclient.jar:?]
      at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:155) [libwstclient.jar:?]
      at com.vmware.vsm.utils.ServiceUtil.getSamlTokenForSvcAccount(ServiceUtil.java:138) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.loadSamlTokenAndPrivateKey(VCenterListener.java:473) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.initializeConnection(VCenterListener.java:268) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.run(VCenterListener.java:301) [vsm.jar:?]
2023-11-02T23:11:57.843-04:00 INFO [Thread-4] ServiceUtil.java 142 - Password expired. Resetting service account password

OVF deployment may hang at power on

In /var/log/vmware/vpxd/vpxd.log you will see events similar to below

2024-02-21T18:34:51.721-06:00 info vpxd[27763] [Originator@6876 sub=VmProv opID=TxId: ########-####-####-####-########51b0-47-01] P
owering on VM '[vsanDatastore] ########-####-####-####-########3960/Interconnect-C3-IX-R1.vmx' on host <host fqdn/ip>
2024-02-21T18:34:51.725-06:00 warning vpxd[27763] [Originator@6876 sub=vmomi.soapStub[258] opID=TxId: ########-####-####-####-####
###51b0-47-01] SOAP request returned HTTP failure; <<io_obj p:0x00007fdab0421b10, h:182, <TCP '127.0.0.1 : 37532'>, <TCP '127.0.0.1
: 15007'>>, /vsm/ovfConsumer/>, method: notifyPowerOn; code: 500(Internal Server Error)
2024-02-21T18:34:51.725-06:00 info vpxd[27763] [Originator@6876 sub=OvfConsumers opID=TxId: ########-####-####-####-########51b0-47
-01] Failed to invoke OVF stub adapter, will re-try after login; N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.S
ecurityError
--> )
--> [context]zKq7AVECAQAAAIUcWQEsdnB4ZAAAxbVTbGlidm1hY29yZS5zbwAAUglDAIwxRACaSEsBoy8XbGlidm1vbWkuc28AAX6eJQFZICABOk4gAZ3HH4JJ8TUBdn
B4ZACCscNFAoKExUUCAX02GoNhyScBbGlidmltLXR5cGVzLnNvAIJGprsBgrXTuwGCNIHUAYLM9SACgn+wGwKC670kAoJI3SQCgkr9JAKCfQhiAoLPGGICgvkvYQKCUothA
oLJH9QBgk0M1QGChw3VAYJTDtUBgrIO1QGD5v4xAYK5d2MCAdXDG4IgokMCgn0IYgKCzxhiAoL5L2ECgpQCYgIA5ss3APkkOACTwFEEro4AbGlicHRocmVhZC5zby4wAAUv
3g9saWJjLnNvLjYA[/context]

At the same time in /var/log/vmware/vsm/vsm.log

2024-02-21T18:34:51.724-06:00 ERROR [pool-5-thread-1] VsmActivationValidator.java 267 - Failed to validate user: only vpxd-svc-acct requests allowed and not

Environment

VMware vCenter Server 8.0.2

Cause

The root cause of the problem is missing jar files from the classpath of the VSM service.

Resolution

This has been fixed in vCenter 8.0 U2b.

VMware vCenter Server 8.0 Update 2b Release Notes

Workaround:
As a workaround, restart the VSM service to recreate the service account.

SSH to vCenter via root
Restart the VSM service:
```
# service-control --restart vsm
```

OR

Login to the VAMI of vCenter and restart the VMware vService Manager

Note: Every time the VSM service is restarted, a new service account and password is created.