VCD Task operations takes longer time to complete in a multiple VMware Cloud Director Cell environment

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• VM Management Tasks such as deploying, powering on and powering off from vCloud Director take a much longer time compared to direct operations via vCenter Server.
• Tasks take 20-30 minutes to complete in vCloud
• Observe "isHealthy": "false" in the jmx.log for Group": {

 "@key": "type",
 "@value": "ActiveMQ", 

• Observe "This cell cannot send messages to cell" in server-group-communications.log
• Observe below logs where vCD receives task update from VC. 

In logs, you would observe that task successful update has already received, when searched using task moref on cell which is running the listener but this cell is still waiting and receives the update after polling VC directly.

In below ex: We can observe that Task success event has reached in VC at 12:54 on another cell but cell that triggered and waiting, received completion event by polling VC.

-----
2021-MM-DD 12:54:11,XXX | DEBUG    | Thread-XXXX (ActiveMQ-client-global-threads) | TaskManager                    | Handling completion update from MessageBusAdapter for task ... with state SUCCESS |

2021-MM-DD 12:58:22,XXX | DEBUG    | VC.TaskManager.TaskCompletionsRetriever | WaitHandle                     | Task updated. task ... state = SUCCESS ...
-----

• Run below command in all the cells, If you observe output is empty. It means Artemis clustering is broken.

netstat -tanp | grep ESTABLISHED | tr -s " "| cut -d " " -f 5 | grep 61616 | sort | uniq

Environment

VMware Cloud Director 10.x

Cause

This issue is caused by Active MQ Artemis clustering communication problems between active vCloud Director Cells.

Resolution

• To resolve this issue:

1. Restart all cells
2. Upgrade to vCD 10.3.2.1

 

• This issue is resolved in VMware Cloud Director 10.3.2.1