VM operations hang due to expired VSM service account password

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

On vCenter Server 8.0U2, when attempting to perform VM operations, the operation hangs and completes after an unknown amount of time or fails to complete.

Examples of affected VM operations:
- Powering on
- Rebooting
- Cloning
- Deployment of VMs via OVA/OVF

In some cases an error message is shown saying which can be seen in Events.

Failed to clone state for the entity '<VM_Template_Name>' on extension vService Manager

In /var/log/vmware/vsm/vsm.log you may see:

<YYYY-MM-DD>T<time> INFO [Thread-4] ServiceUtil.java 137 - Acquiring SAML token for user vmware-vsm-########-####-####-####-########acba@vsphere.local 
<YYYY-MM-DD>T<time> ERROR [Thread-4] SoapBindingImpl.java 185 - SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-vsm-########-####-####-####-########acba, Domain: vsphere.local} Please see the server log to find more detail regarding exact cause of the failure.
..
com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983) [libwstclient.jar:?]
      at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) [libwstclient.jar:?]
      at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:155) [libwstclient.jar:?]
      at com.vmware.vsm.utils.ServiceUtil.getSamlTokenForSvcAccount(ServiceUtil.java:138) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.loadSamlTokenAndPrivateKey(VCenterListener.java:473) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.initializeConnection(VCenterListener.java:268) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.run(VCenterListener.java:301) [vsm.jar:?]
<YYYY-MM-DD>T<time>  INFO [Thread-4] ServiceUtil.java 142 - Password expired. Resetting service account password

OVF deployment may hang at power on.

In /var/log/vmware/vpxd/vpxd.log you will see events similar to below

<YYYY-MM-DD>T<time> info vpxd[27763] [Originator@6876 sub=VmProv opID=TxId: ########-####-####-####-########51b0-47-01] P
owering on VM '[vsanDatastore] ########-####-####-####-########3960/Interconnect-##-IX-##.vmx' on host <host fqdn/ip>
<YYYY-MM-DD>T<time> warning vpxd[27763] [Originator@6876 sub=vmomi.soapStub[258] opID=TxId: ########-####-####-####-####
###51b0-47-01] SOAP request returned HTTP failure; <<io_obj p:0x00007fdab0421b10, h:182, <TCP '127.0.0.1 : 37532'>, <TCP '127.0.0.1
: 15007'>>, /vsm/ovfConsumer/>, method: notifyPowerOn; code: 500(Internal Server Error)
<YYYY-MM-DD>T<time> info vpxd[27763] [Originator@6876 sub=OvfConsumers opID=TxId: ########-####-####-####-########51b0-47
-01] Failed to invoke OVF stub adapter, will re-try after login; N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.S
ecurityError
--> )
--> [context]zKq7AVECAQAAAIUcWQEsdnB4ZAAAxbVTbGlidm1hY29yZS5zbwAAUglDAIwxRACaSEsBoy8XbGlidm1vbWkuc28AAX6eJQFZICABOk4gAZ3HH4JJ8TUBdn
B4ZACCscNFAoKExUUCAX02GoNhyScBbGlidmltLXR5cGVzLnNvAIJGprsBgrXTuwGCNIHUAYLM9SACgn+wGwKC670kAoJI3SQCgkr9JAKCfQhiAoLPGGICgvkvYQKCUothA
oLJH9QBgk0M1QGChw3VAYJTDtUBgrIO1QGD5v4xAYK5d2MCAdXDG4IgokMCgn0IYgKCzxhiAoL5L2ECgpQCYgIA5ss3APkkOACTwFEEro4AbGlicHRocmVhZC5zby4wAAUv
3g9saWJjLnNvLjYA[/context]

At the same time in /var/log/vmware/vsm/vsm.log

<YYYY-MM-DD>T<time> ERROR [pool-5-thread-1] VsmActivationValidator.java 267 - Failed to validate user: only vpxd-svc-acct requests allowed and not

Environment

vCenter Server 8.0U2

Cause

The root cause of the problem is missing jar files from the classpath of the VSM service.

Resolution

This issue has been resolved in VMware vCenter Server 8.0 Update 3b release. To download go to Download Broadcom products and software

Workaround:
As a workaround, restart the VSM service to recreate the service account.

SSH to vCenter via root
Restart the VSM service:
```
# service-control --restart vsm
```

OR

Login to the VAMI of vCenter and restart the VMware vService Manager

Note: Every time the VSM service is restarted, a new service account and password is created.