Cloud Director VM tasks are slow or stall out in Cloud Director
search cancel

Cloud Director VM tasks are slow or stall out in Cloud Director


Article ID: 325520


Updated On:


VMware Cloud Director


  • VM tasks are slow or stall out.
  • /opt/vmware/vcloud-director/logs/vcloud-container-debug.log may show messages similar to:
2023-01-15 10:41:19,542 | WARN | vim-proxy-activity-pool-68 | PowerOnVmActivity | [Activity Execution] Timed out while waiting for VMs [[vcId=<VC_UUID>, moref=vm-372331]] status to change to powered on in the inventory cache. Waited 120 seconds for the VMs status to change to powered on. ...
  • /opt/vmware/vcloud-director/logs/jms-debug.log may show messages similar to:
2023-01-30 11:24:54,390 | DEBUG | org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@38a1c423) | ServerConsumerImpl | ServerConsumerImpl [id=0, filter=null, binding=LocalQueueBinding [address=VC_LISTENER_SEPA_1, queue=QueueImpl[name=<LISTENER_UUID>, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=randomBroker9345], temp=true]@ce6fedc, filter=null, name=<LISTENER_UUID>, clusterName=<CLUSTER_UUID>]] is busy for the lack of credits. Current credits = -64433 Can't receive reference Reference[240455]:NON-RELIABLE:LargeServerMessage[messageID=240455,durable=false,userID=<USER_UUID>,priority=4, timestamp=Mon Jan 30 11:24:54 EST 2023,expiration=Mon Jan 30 11:25:24 EST 2023, durable=false, address=VC_LISTENER_SEPA_1, properties=TypedProperties[__AMQ_CID=<AMQ_UUID>,_AMQ_ROUTING_TYPE=0,_AMQ_LARGE_SIZE=600115]]@324502783 |.


VMware Cloud Director 10.x


This is caused by a defect in the message bus expiration for larger messages.


This issue is fixed in Cloud Director

If an upgrade of Cloud Director is not possible, a workaround to change the following configuration parameters can be attempted.
  1. Take a backup of Cloud Director before proceeding. For Appliance based Cloud Director Cells see the VMware Cloud Director documentation, Backup and Restore of VMware Cloud Director Appliance.
  2. SSH to the Primary Cell as root.
  3. Update the following configuration parameters using the Cell Management Tool.
    NOTE: the config updates only need to be run once from one Cell which will place them in the Cloud Director database.

    /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vc-task-completions-retrieval-timer-interval-sec -v 10
    /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.activities.activityRelayPollingIntervalMs -v 10000

  4. Stop and Start the Cloud Director service on the Primary Cell and all other Cells in the cluster.
    Each Cell reads the updated configuration parameter from the Cloud Director database on startup.
    NOTE: do not restart the entire Cloud Director Cell VM, only the Cloud Director services need to be restarted.

    Stop the Cloud Director Services:
    /opt/vmware/vcloud-director/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -s
    systemctl stop vmware-vcd

    Start the Cloud Director Services:
    systemctl start vmware-vcd
  5. To revert the settings to default after upgrading Cloud Director or at any other time, delete the custom configuration parameters using the Cell Management Tool from one Cell and restart the Cloud Director services on all Cells.

    /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vc-task-completions-retrieval-timer-interval-sec -d
    /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.activities.activityRelayPollingIntervalMs -d


Additional Information

VMware Cloud Director Cell Management Tool Reference