vApp deployment via template delays at the final stage of booting.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• At the final stage of vApp deployment via Vm template, the deployment process might take up up to an hour.
• Through vSphere Client UI, the deployed VM might seem functional except all the power options being greyed out.
• The customer might have to manually power on the VM.
• Host management logs will not indicate any sign of task delay.

var/log/vmware/vpxd/vpxd.log shows following pattern..

[YYYY-MM-DDTHH:MM:SS] error vpxd[776988] [Originator@6876 sub=IO.Http opID=ls2x1zwr-343159-auto-7cs8-h5:70069383-56-01] User agent failed to send request; (null), N7Vmacore16TimeoutExceptionE(Operation timed out) --> [context]zKq7AVECAQAAAIUcWQEWdnB4ZAAAxbVTbGlidm1hY29yZS5zbwAAUglDAIwxRADMSksAa5dIAbxNIGxpYnZtb21pLnNvAAGdxx8BfTYaAoRUAmxpYnZzbS10eXBlcy5zbwCD4b5FAnZweGQAg3PDRQKD9sNFAoOExUUCAX02GgHNKhqEA88mAWxpYnZpbS10eXBlcy5zbwCDoqG7AQDmyzcA+SQ4AJPAUQWujgBsaWJwdGhyZWFkLnNvLjAABi/eD2xpYmMuc28uNgA=[/context]
[YYYY-MM-DDTHH:MM:SS] error vpxd[76811] [Originator@6876 sub=OvfConsumers opID=ls2x1zwr-343159-auto-7cs8-h5:70069383-56-01] Failed to authenticate with VSM; stubAdapter: <<cs p:00007f864c1ee4f0, TCP:localhost:1080>, /vsm/sdk>, N7Vmacore16TimeoutExceptionE(Operation timed out)

• vsm.log on NSX Manager shows following pattern.

[YYYY-MM-DDTHH:MM:SS] ERROR [Thread-4] SoapBindingImpl.java 185 - SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-vsm-.., Domain: vsphere.local} Please see the server log to find more detail regarding exact cause of the failure.
        at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178) ~[?:1.8.0_362]
        at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116) ~[?:1.8.0_362]
        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259) ~[?:1.8.0_362]
        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289) ~[?:1.8.0_362]
        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:182) [libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:116) [libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983) [libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) [libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:155) [libwstclient.jar:?]
        at com.vmware.vsm.utils.ServiceUtil.getSamlTokenForSvcAccount(ServiceUtil.java:138) [vsm.jar:?]
        at com.vmware.vsm.vc.VCenterListener.loadSamlTokenAndPrivateKey(VCenterListener.java:473) [vsm.jar:?]
        at com.vmware.vsm.vc.VCenterListener.initializeConnection(VCenterListener.java:268) [vsm.jar:?]
        at com.vmware.vsm.vc.VCenterListener.run(VCenterListener.java:301) [vsm.jar:?] 
[YYYY-MM-DDTHH:MM:SS]  INFO [Thread-4] ServiceUtil.java 142 - Password expired. Resetting service account password
... 
[YYYY-MM-DDTHH:MM:SS] ERROR [pool-5-thread-1] VsmActivationValidator.java 267 - Failed to validate user: only vpxd-svc-acct requests allowed and not

Environment

VMware vCenter Server 8.0.2

Cause

• vApp requires several vServices to be provided from vCenter.
• There vServices undergo password authentication when communicating with other vCenter services.
• These passwords are frequently updated by vService Management service (vsm).
• vsm might fall into hanging status during vService password update.
• Deployment process freezes since vpxd cannot communicate with vServices that handles vApp power cycles.

Resolution

This issue will be resolved in vCenter Server 8.0.U2b.

Workaround:

Restarting vsm service will break the hung process.

service-control --restart vsm