Data collection for vCenter Server Cloud Account hangs at 'Collecting data...' or fails with 'Failed to retrieve properties.' in VMware Aria Automation
search cancel

Data collection for vCenter Server Cloud Account hangs at 'Collecting data...' or fails with 'Failed to retrieve properties.' in VMware Aria Automation

book

Article ID: 446826

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • In Aria Automation Assembler, the data collection status for the vCenter Account (located under Infrastructure > Connections > Cloud Accounts > vCenter Account) either hangs at 'Collecting data...' or fails with the message 'Failed to retrieve properties.'





  • The following log entries can be observed in /var/log/services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log.

    The vSphere enumeration task stalls for one hour after the following logs appear:

    YYYY-mm-ddT01:51:43.###Z INFO provisioning [host='provisioning-service-app-##########-####' thread='vsphere-enum-##' user='' org='' trace='' parent='' span=''] 	c.v.p.c.m.a.v.e.StoragePolicyEnumerationService.enumerate:88 - [<vCenter FQDN>]: Begin storage policy enumeration.
YYYY-mm-ddT01:51:43.###Z INFO provisioning [host='provisioning-service-app-##########-####' thread='vsphere-enum-##' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.r.pbm.PbmSession.<init>:63 - 	Creating PBM session using client [com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.pbm.PbmClient@#######], session [	com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.pbm.PbmSession@########]

    Subsequently, the following error log is recorded:

    YYYY-mm-ddT02:51:46.###Z ERROR provisioning [host='provisioning-service-app-##########-####' thread='vsphere-enum-##' user='' org='' trace='' parent='' span=''] 	c.v.p.c.m.a.v.r.vc.PcDomain.executeRetrievePropertiesExCall:468 - [session[########-####-####-####-############]########-####-####-####-############]: Failed to retrieve properties. 	com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: The object 	'vmodl.query.PropertyCollector:session[########-####-####-####-############]########-####-####-####-############' has already been deleted or has not been completely created Please see the server log to find 	more detail regarding exact cause of the failure.
                at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:158)
                at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:112)
                at com.sun.xml.ws.client.sei.StubHandler.readResponse(StubHandler.java:225)
                at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:176)
                at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:263)
                at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
                at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:62)
                at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:132)
                at jdk.proxy2/jdk.proxy2.$Proxy395.retrievePropertiesEx(Unknown Source)
                at com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.vc.PcDomain.executeRetrievePropertiesExCall(PcDomain.java:448)
                at com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.vc.PcDomain.handleRetrieveResults(PcDomain.java:368)
                at com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.vc.PcDomain.lambda$retrieveProperties$0(PcDomain.java:93)
                at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:757)
                at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:735)
                at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2182)
                at com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.vc.PcDomain.retrieveProperties(PcDomain.java:91)
                at com.vmware.photon.controller.model.adapters.vsphere.remotecallmanager.vc.PcDomain.retrieveProperties(PcDomain.java:69)
                at com.vmware.photon.controller.model.adapters.vsphere.enumeration.VsphereEnumerationAdapterService.getDatacenters(VsphereEnumerationAdapterService.java:683)
                at java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)
                at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
                at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
                at com.vmware.photon.controller.model.adapters.vsphere.executor.ContextWrapperUtil.lambda$wrapWithOperationContext$0(ContextWrapperUtil.java:24)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
                at java.base/java.lang.Thread.run(Thread.java:840)

Environment

VMware Aria Automation 8.18.x

Cause

This issue is caused by a known bug in the JDK ForkJoinPool (JDK-8330017). This bug causes the storage profile retrieval process to stall during the enumeration task.

Resolution

This issue is resolved in VMware Aria Automation 8.18.1 Patch 5, which includes an updated Java version that addresses the bug.

To work around this issue, restart the provisioning-service-app pod

  1. Delete the pod to restart the service:
    kubectl delete pod -n prelude -l app=provisioning-service-app

  2. Run the following command to verify that the pod is running:
    kubectl get pods -n prelude -l app=provisioning-service-app -w

Powered by Wolken Software