Deployment request is hung at create in progress 7/9 and never times out or fails

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• The deployment is hung at provisioning stage and does not times out even after many hours. We do not see the machine getting deployed in the Vcenter.
  
  
• When the request is cancelled and resubmitted it works fine
• Other requests completes fine with no errors
  
  
  The provisioning logs shows us the below exception
  
  Log location: /services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log
  
  YYYY-MM-DDT##:##:##.###Z INFO provisioning [host='provisioning-service-app-123456789' thread='vsphere-io-277' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.r.v.VSphereConnectionPool.getConnection:90 - Found existing connection. connectionProperty:[class  VSphereConnectionProperties {
  
  Few seconds later, there was a socket exception for the target VC
  
  YYYY-MM-DDT##:##:##.###Z ERROR provisioning [host='provisioning-service-app-123456789' thread='vsphere-io-277' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.r.vc.VcSession.lambda$getServiceContent$19:591 - [https://<VC_FQDN>/sdk]: Failed to retrieve service content [com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.SocketTimeoutException: Connect timed out
        at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:103)
      com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.vc.VcSession.lambda$getServiceContent$19(VcSession.java:589)
    Caused by: java.net.SocketTimeoutException: Connect timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:551)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633

Environment

Aria automation 8.x

Cause

The request is processed for provisioning where it goes to find VC connection and finds an existing connection. After a while it follows with a socket exception due temporary infra(network) glitch lead to the stuck deployment

Resolution

In future versions there will be better error handling so that in case of exception it fails so that internally the VSphereConnectionPool goes to create a new connection instead of getting hung.



Workaround

Cancel and re-submit the deployment request.