VMware Cloud Director AMQP Broker vhost configuration not updating after re-initialization in VMware Cloud Director

Products

VMware Cloud Director

Issue/Introduction

After updating the AMQP vhost via the VMware Cloud Director API, there still appear to be attempts to connect to the previously configured vhost.
The previous vhost was no longer valid prior to the change in configuration.
After the change was performed, logs indicate that AMQP connections are starting without issue, as in the following example from /opt/vmware/vcloud-director/logs/obj-ext-debug.log:

2026-07-02 04:55:00,443 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | AMQP settings have changed. Old: [host=###.###.###, port=####, username=####, exchange=########, vHost=####, useSSL=true, prefix=###]. New: [host=###.###.###, port=####, username=####, exchange=########, vHost=###, useSSL=true, prefix=###]. | 2026-07-02 04:55:00,522 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Reconnecting listener for queueName: [Ljava.lang.String;@########. Currently running status: false | 2026-07-02 04:55:00,523 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Creating AMQP connection and listeners: [host=###.###.###, port=####, username=####, exchange=########, vHost=####, useSSL=true, prefix=###] |
2026-07-02 04:55:00,523 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Destroying connection factory for AMQP host: ###.###.### |
2026-07-02 04:55:00,523 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Creating connection factory |
2026-07-02 04:55:00,525 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Creating new rabbit template for AMQP host: v###.###.### |
2026-07-02 04:55:00,525 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | New rabbit template created for AMQP host: ###.###.### |
2026-07-02 04:55:00,525 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Declaring and establishing listeners for reply_to queue: vcd.ext.replyQueue.########-####-####-####-############ |
2026-07-02 04:55:00,610 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | New AMQP connection created |
2026-07-02 04:55:00,634 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Declared reply_to queue: vcd.ext.replyQueue.########-####-####-####-############ |
2026-07-02 04:55:00,634 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Establishing listeners for reply_to queue: vcd.ext.replyQueue.########-####-####-####-############ |
2026-07-02 04:55:00,635 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Completed setup of reply_to queue and listener: vcd.ext.replyQueue.########-####-####-####-############ |
2026-07-02 04:55:00,635 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Starting listener for queueNames: [vcd.ext.replyQueue.########-####-####-####-############] |
2026-07-02 04:55:00,635 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Starting listener for queue: vcd.ext.replyQueue.########-####-####-####-############ |
2026-07-02 04:55:00,683 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | Started listener for queue: vcd.ext.replyQueue.########-####-####-####-############ |
2026-07-02 04:55:00,683 | DEBUG | pool-28-thread-1 | AmqpMessageManagerImpl | AMQP connection refresh completed |
2026-07-02 05:09:44,092 | DEBUG | SimpleAsyncTaskExecutor-3 | AmqpMessageManagerImpl | AMQP connection closed |
2026-07-02 05:10:39,505 | DEBUG | SimpleAsyncTaskExecutor-14 | AmqpMessageManagerImpl | New AMQP connection created |
In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, there are periodic errors indicating that a previous failed connection is being recovered unsuccessfully:

2026-07-07 12:28:44,203 | ERROR | AMQP Connection ###.###.###.###:#### | ForgivingExceptionHandler | Caught an exception during connection recovery! |
java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:127)
...
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method<connection.close>(reply-code=530, reply-text=NOT_ALLOWED - vhost ###### not found, class-id=10, method-id=40)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:451)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:268)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:139)
... 10 more
2026-07-07 12:28:44,223 | ERROR | RabbitMQ Error On Write Thread | ForgivingExceptionHandler | An unexpected connection driver error occured |
java.net.SocketException: Connection or outbound has closed
...

Environment

VMware Cloud Director 10.6.x

Cause

An existing RabbitMQ client process is attempting to recover and not exiting independently.

Resolution

To clear this outstanding process, it would be recommended to restart the VMware Cloud Director (VCD) service on all cells encountering the AMQP errors:

Log in to the target VCD cell via SSH or console as the root user.
Perform the following to safely stop the VCD service using the following command:

/opt/vmware/vcloud-director/bin/cell-management-tool -u <admin_username> cell --shutdown
Restart the VCD service:

systemctl restart vmware-vcd
Confirm that the vCD service is running:

systemctl status vmware-vcd

Additional Information

Note: AMQP-based functionality is deprecated starting in version 10.6 of VMware Cloud Director and it is recommended to use the MQTT-backed functionality, as discussed in the following:

Install and Configure a RabbitMQ AMQP Broker for VMware Cloud Director

For more information regarding the management of the VMware Cloud Director service, refer to the following:

Steps to stop, start, or restart the VMware Cloud Director service before rebooting the nodes