VM operations hang due to expired VSM service account password
search cancel

VM operations hang due to expired VSM service account password

book

Article ID: 316456

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

On vCenter Server 8.0U2, when attempting to perform VM operations, the operation hangs and completes after an unknown amount of time or fails to complete.

  • Examples of affected VM operations:
    • Powering on
    • Rebooting
    • Cloning
    • Deployment of VMs via OVA/OVF

  • In some cases, vSphere client might report an error in the 'Events' such as:
          Failed to clone state for the entity '<VM_Template_Name>' on extension vService Manager

  • The following errors are captured in the vCenter vsm.log (/var/log/vmware/vsm/vsm.log):  

    [timestamp] INFO [Thread-4] ServiceUtil.java 137 - Acquiring SAML token for user vmware-vsm-########-####-####-####-########@example.domain 
    [timestamp] ERROR [Thread-4] SoapBindingImpl.java 185 - SOAP fault
    com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-vsm-########-####-####-####-########, Domain: example.domain} Please see the server log to find more detail regarding exact cause of the failure.
    ..
    com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983) [libwstclient.jar:?]
          at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) [libwstclient.jar:?]
          at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:155) [libwstclient.jar:?]
          at com.vmware.vsm.utils.ServiceUtil.getSamlTokenForSvcAccount(ServiceUtil.java:138) [vsm.jar:?]
          at com.vmware.vsm.vc.VCenterListener.loadSamlTokenAndPrivateKey(VCenterListener.java:473) [vsm.jar:?]
          at com.vmware.vsm.vc.VCenterListener.initializeConnection(VCenterListener.java:268) [vsm.jar:?]
          at com.vmware.vsm.vc.VCenterListener.run(VCenterListener.java:301) [vsm.jar:?]
    [timestamp]  INFO [Thread-4] ServiceUtil.java 142 - Password expired. Resetting service account password

  • OVF deployment may hang at power on.

    • The following errors are captured in the vCenter vpxd.log (/var/log/vmware/vpxd/vpxd.log)

[timestamp] info vpxd[27763] [Originator@6876 sub=VmProv opID=TxId: ########-####-####-####-########51b0-47-01] P
owering on VM '[vsanDatastore] ########-####-####-####-########/Interconnect-##-IX-##.vmx' on host <host fqdn/ip>
[timestamp] warning vpxd[27763] [Originator@6876 sub=vmomi.soapStub[258] opID=TxId: ########-####-####-####-####
###51b0-47-01] SOAP request returned HTTP failure; <<io_obj p:0x00007fdab0421b10, h:182, <TCP '127.0.0.1 : 37532'>, <TCP '127.0.0.1
 : 15007'>>, /vsm/ovfConsumer/>, method: notifyPowerOn; code: 500(Internal Server Error)
[timestamp] info vpxd[27763] [Originator@6876 sub=OvfConsumers opID=TxId: ########-####-####-####-########] Failed to invoke OVF stub adapter, will re-try after login; N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.S
ecurityError
--> )
--> [context]zKq7AVECAQAAAIUcWQEsdnB4ZAAAxbVTbGlidm1hY29yZS5zbwAAUglDAIwxRACaSEsBoy8XbGlidm1vbWkuc28AAX6eJQFZICABOk4gAZ3HH4JJ8TUBdn
B4ZACCscNFAoKExUUCAX02GoNhyScBbGlidmltLXR5cGVzLnNvAIJGprsBgrXTuwGCNIHUAYLM9SACgn+wGwKC670kAoJI3SQCgkr9JAKCfQhiAoLPGGICgvkvYQKCUothA
oLJH9QBgk0M1QGChw3VAYJTDtUBgrIO1QGD5v4xAYK5d2MCAdXDG4IgokMCgn0IYgKCzxhiAoL5L2ECgpQCYgIA5ss3APkkOACTwFEEro4AbGlicHRocmVhZC5zby4wAAUv
3g9saWJjLnNvLjYA[/context]

    • The following errors are captured in the vCenter vsm.log (/var/log/vmware/vsm/vsm.log)

[timestamp] ERROR [pool-5-thread-1] VsmActivationValidator.java 267 - Failed to validate user: only vpxd-svc-acct requests allowed and not

Environment

vCenter Server 8.0 Update 2

Cause

The root cause of the problem is missing jar files from the classpath of the VSM service.

Resolution

This issue has been resolved in VMware vCenter Server 8.0 Update 3b release. To download go to Download Broadcom products and software

As a workaround, restart the VSM service to recreate the service account. 

  1. SSH to vCenter via root
  2. Restart the VSM service:
    service-control --restart vsm
OR
  1. Login to the vCenter Appliance Management Interface (VAMI) as root: https://<vCenter_FQDN_or_IP_ADDRESS>:5480
  2. Go to Services
  3. Select the VMware vService Manager and click Restart

Note: Every time the VSM service is restarted, a new service account and password is created.
Powered by Wolken Software