"Managing Depots is temporarily unavailable. Could not retrieve health status for Auto Deploy and Image Builder services" error in AutoDeploy Software Depots page in the vCenter
search cancel

"Managing Depots is temporarily unavailable. Could not retrieve health status for Auto Deploy and Image Builder services" error in AutoDeploy Software Depots page in the vCenter

book

Article ID: 387775

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • While navigating in the AutoDeploy Software Depots, Deploy Rules, intermittently we see the below error:



  • Sometimes another error can be received "Managing Rules is temporarily unavailable. Could not retrieve health status for Auto Deploy and Image Builder services. One or both services may be down or unreachable"
  • While checking the service-control --status --all, we see that vmware-imagebuilder and vmware-rbd-watchdog is up and running.
  • Applying/Editing an Auto Deploy rule might get stuck at 2%.
  • Upon checking the /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log:

    [YYYY-MM-DDTHH:MM:SS] [ERROR] http-nio-5090-exec-2630 70830508 114081 200958 com.vmware.vise.actionsfw.impl.ActionsEvaluator
       An error occurred while retrieving data for actions evaluation. com.vmware.vise.data.query.internal.DataServiceException: The query execution timed out because of a back-end data adapter 'com.vmware.vsphere.client.h5.autodeploy.impl.DepotDataProviderAdapter' which took more than 120 seconds.
            at com.vmware.vise.data.query.impl.DataAdapterUtil.processDataAdapterTaskException(DataAdapterUtil.java:159)
            at com.vmware.vise.data.query.impl.DataAdapterUtil.executeAdapterTasks(DataAdapterUtil.java:125)
            at com.vmware.vise.data.query.impl.DataAdapterRetriever.runDataTasks(DataAdapterRetriever.java:374)
            at com.vmware.vise.data.query.impl.DataAdapterRetriever.invokeDataAdapters(DataAdapterRetriever.java:325)
            at com.vmware.vise.data.query.impl.DataAdapterRetriever.getResultsFromDataAdapters(DataAdapterRetriever.java:78)
            at com.vmware.vise.data.query.impl.DataManager.getResultsFromDataAdapters(DataManager.java:175)
            at com.vmware.vise.data.query.impl.DataServiceImpl.getResults(DataServiceImpl.java:319)
            at com.vmware.vise.data.query.impl.DataServiceImpl.getResponse(DataServiceImpl.java:201)
            at com.vmware.vise.data.query.impl.DataServiceImpl.getData(DataServiceImpl.java:175)
            at com.sun.proxy.$Proxy199.getData(Unknown Source)
            at com.vmware.vise.data.adapters.risetovise.RiseToViseQueryExecutor.getData(RiseToViseQueryExecutor.java:329)
            at com.vmware.vise.data.adapters.risetovise.RiseToViseQueryExecutor.executeComplexQueriesInVise(RiseToViseQueryExecutor.java:266)
            at com.vmware.vise.data.adapters.risetovise.RiseToViseQueryExecutor.executeComplexQueries(RiseToViseQueryExecutor.java:194)
            at com.vmware.vise.data.adapters.risetovise.RiseToViseQueryExecutor.executeQueryBatch(RiseToViseQueryExecutor.java:139)
           at com.vmware.vise.data.adapters.risetovise.RiseToViseQueryServiceFactory$RiseToViseQueryBatchService.executeQueryBatch(RiseToViseQueryServiceFactory.java:142)
    at com.vmware.vsphere.bridge.DenyConfigurationFilesFilter.doFilter(DenyConfigurationFilesFilter.java:45)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:181)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
            at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1698)
            at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
            at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
            at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
            at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
            at java.lang.Thread.run(Thread.java:750)
    Caused by: java.util.concurrent.TimeoutException: null
            at com.vmware.vise.util.concurrent.ExecutorUtil.executeTasks(ExecutorUtil.java:744)
            at com.vmware.vise.data.query.impl.ProfiledTaskExecutor.executeTasks(ProfiledTaskExecutor.java:70)
            at com.vmware.vise.data.query.impl.DataAdapterUtil.executeAdapterTasks(DataAdapterUtil.java:112)
  • Under /var/log/vmware/vsphere-ui/logs/dataservice.log:

    [YYYY-MM-DDTHH:MM:SS] [WARN ] http-nio-5090-exec-2624 70830555 114081 200958 ProfilingLogService [queryId: q-4735994:h5c-DataAccessController] The Data Service detected slow execution:
    The query batch execution took too long: 120006 milliseconds.

    The queries in the batch sent to the Data Service are:
    Query[0]:

    QuerySpec
     QueryName: q-4735994:h5c-DataAccessController
     ResourceSpec
      Constraint: RelationalConstraint
       TargetType: imagebuilder:Depot
       Relationship: imageBuilderDepots
       IsSourceConstrained: false
       Constraint: ObjectIdentityConstraint
        TargetType: null
       Target: ManagedObjectReference: type = Folder, value = group-d1, serverGuid = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx
      PropertySpec[1]
       ProviderType: imagebuilder:Depot
       Relationship: null
       Properties[5]
        PropertyName: name
        PropertyName: depotType
        PropertyName: url
        PropertyName: status
        PropertyName: lastUpdated
     ResultSpec:
      Offset: null
      MaxResultCount: null
      OrderingCriteria
       OrderPropertySpec[1]
        ProviderType: Object
        Relationship: null
        SortType: ASCENDING
        OrderingProperties[1]
         PropertyName: id
     Options:
      VISE_TO_RISE_TRANSLATOR_STRICT_MODE=true

  • Under /var/log/vmware/vsphere-ui/logs/opId.log:

    [YYYY-MM-DDTHH:MM:SS] [ERROR] data-service-pool-786 70830555 114081 200958 Invocation of 'com.vmware.vim.binding.imagebuilder.DepotSpec[] com.vmware.vim.binding.imagebuilder.DepotManager.GetDepots(com.vmware.vim.binding.imagebuilder.DepotManager$DepotFilterSpec)' to http://localhost:1080/external-vecs/http1/<vc-fqdn>/443/vmw/imagebuilder/vmomi for opId 'q-4735994:h5c-DataAccessController:DepotDataProviderAdapter:10075188-54h1y-h5:70830555' failed in ? ms, error: com.vmware.vim.vmomi.client.exception.ConnectionException:http://localhost:1080/external-vecs/http1/<vc-fqdn>/443/vmw/imagebuilder/vmomi invocation failed with "java.net.SocketTimeoutException: Read timed out"

  • There are no such related errors/warnings in '/var/log/vmware/vmware-imagebuilder/main.log' at the time of issue.

Environment

vCenter 7.x.x

vCenter 8.x.x

Cause

Connection pool size for the Image Builder service reaches the maximum limit and gets exhausted.

Resolution

Note: Please note that the above change will result in ImageBuilder service consuming around 1.5GB of memory so please validate the memory usage on the vCenter before performing the below steps

Take a valid snapshot/backup of the vCenter Appliance before proceeding with the below steps:

Follow the below steps:

    1. Login to vCenter using a SSH session via putty.
      • cd /usr/lib/vmware-imagebuilder/site-packages/imagebuilder
    2. Take a copy of the 'ibserver.py' file.
      • cp ibserver.py /storage/core/ibserver.py.bkp
    3. Open the file using a vi editor:
      • vi ibserver.py
      • In vCenter 7.x.x
        1. Look for the parameter 'self.soapHttpd.SetMaxWorkers(8)'.
          1. Change the value from (8) to (128) for self.soapHttpd.SetMaxWorkers(8).
          2. soapHttpd.SetMaxWorkers(8) >>>> self.soapHttpd.SetMaxWorkers(128)
      • In vCenter 8.x.x
        1. Look for the parameter 'pyVmomiServerOptions.max_workers = 8'.
          1. Change the value from 8 to 128 for pyVmomiServerOptions.max_workers = 8.
          2. pyVmomiServerOptions.max_workers = 8 >>>> pyVmomiServerOptions.max_workers = 128
    4. Hit ESC, then :wq! to save the file.
    5. Restart vmware-image-builder Service:
      • service-control --restart vmware-imagebuilder
Powered by Wolken Software