Tasks in Cloud Director are taking approximately 10 or 30 minutes to complete.

Products

• VMware Cloud Director

Issue/Introduction

• Tasks in Cloud Director that should take a few seconds are taking longer than usual to complete, in some cases, approximately 10 or 30 minutes.
• Third party software and other services dependent on VCD completing tasks, may timeout.

 

 

Environment

Cloud Director 10.x

Cause

• The Artemis cluster is the internal mechanism used to facilitate cell-to-cell communication.
• In the /opt/vmware/vcloud-director/logs/cell-runtime.log you will see a value in members=x where the expected number is the number of Appliances in the cluster. Anything other than this indicates a cell-to-cell communication issue.
• Activities will be delayed for a period of time, usually around 10 minutes, whilst the system waits for messages to that cell to timeout before proceeding.
• An Activity is a series of steps that work together to complete a task. These steps are called Phases. All the Phases in an Activity are meant to be carried out on one Appliance.
• The other Appliances must be active in the Artemis cluster so that they can send and receive messages to and from the Appliance processing the activity.

Resolution

• To determine which cell is causing the messaging delay, shutdown the VCD services on all nodes other than the Primary.
• If VM or vApp power on and power off tasks initiated in the UI complete quickly, bring up the next node and repeat the same task.
• Continue until the tasks start to become delayed.
• This will indicate which node has an issue and will need redeploying.
• Follow the guidance in the Recover from a VMware Cloud Director Appliance Standby Cell Failure in a High Availability Cluster techdoc to redeploy the node.

Additional Information