Cloud Director VM tasks are slow or stall out in Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• VM tasks are slow or stall out.
• /opt/vmware/vcloud-director/logs/vcloud-container-debug.log may show messages similar to:

2023-01-15 10:41:19,542 | WARN | vim-proxy-activity-pool-68 | PowerOnVmActivity | [Activity Execution] Timed out while waiting for VMs [[vcId=<VC_UUID>, moref=vm-372331]] status to change to powered on in the inventory cache. Waited 120 seconds for the VMs status to change to powered on. ...

• /opt/vmware/vcloud-director/logs/jms-debug.log may show messages similar to:

2023-01-30 11:24:54,390 | DEBUG | org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@38a1c423) | ServerConsumerImpl | ServerConsumerImpl [id=0, filter=null, binding=LocalQueueBinding [address=VC_LISTENER_SEPA_1, queue=QueueImpl[name=<LISTENER_UUID>, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=randomBroker9345], temp=true]@ce6fedc, filter=null, name=<LISTENER_UUID>, clusterName=<CLUSTER_UUID>]] is busy for the lack of credits. Current credits = -64433 Can't receive reference Reference[240455]:NON-RELIABLE:LargeServerMessage[messageID=240455,durable=false,userID=<USER_UUID>,priority=4, timestamp=Mon Jan 30 11:24:54 EST 2023,expiration=Mon Jan 30 11:25:24 EST 2023, durable=false, address=VC_LISTENER_SEPA_1, properties=TypedProperties[__AMQ_CID=<AMQ_UUID>,_AMQ_ROUTING_TYPE=0,_AMQ_LARGE_SIZE=600115]]@324502783 |.

Environment

VMware Cloud Director 10.x

Cause

This is caused by a defect in the message bus expiration for larger messages.

Resolution

This issue is fixed in Cloud Director 10.4.1.1.

Workaround:
If an upgrade of Cloud Director is not possible, a workaround to change the following configuration parameters can be attempted.

1. Take a backup of Cloud Director before proceeding. For Appliance based Cloud Director Cells see the VMware Cloud Director documentation, Backup and Restore of VMware Cloud Director Appliance.
2. SSH to the Primary Cell as root.
3. Update the following configuration parameters using the Cell Management Tool.
   NOTE: the config updates only need to be run once from one Cell which will place them in the Cloud Director database.
   
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vc-task-completions-retrieval-timer-interval-sec -v 10
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.activities.activityRelayPollingIntervalMs -v 10000
    
4. Stop and Start the Cloud Director service on the Primary Cell and all other Cells in the cluster.
   Each Cell reads the updated configuration parameter from the Cloud Director database on startup.
   NOTE: do not restart the entire Cloud Director Cell VM, only the Cloud Director services need to be restarted.
   
   Stop the Cloud Director Services:
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -s
   systemctl stop vmware-vcd
   
   Start the Cloud Director Services:
   systemctl start vmware-vcd
    
5. To revert the settings to default after upgrading Cloud Director or at any other time, delete the custom configuration parameters using the Cell Management Tool from one Cell and restart the Cloud Director services on all Cells.
   
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vc-task-completions-retrieval-timer-interval-sec -d
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.activities.activityRelayPollingIntervalMs -d
    

Additional Information

VMware Cloud Director Cell Management Tool Reference