EBS pods restart and deployments stall after scaling out or replacing nodes in Aria Automation

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

After scaling out a VMware Aria Automation environment from a single node to a cluster, or after replacing existing nodes, event-based functionalities fail.

Specific symptoms include:

• • Deployments stall during the "initialization" phase.
  • Extensibility actions (ABX) and vRO workflows fail to trigger.
  • EBS pods restart repeatedly due to readiness probe timeouts.
  • Logs show the following error: reactor.rabbitmq.RabbitFluxException: Error while creating channel

    Mon Aug 19 04:21:35 AM UTC 2024
    {"status":"UP","details":{"readinessStateHealthIndicator":{"status":"UP"},"rabbitHealthContributor":{"status":"UP","details":{"version":"3.11.4"}},"diskSpaceHealthIndicator":{"status":"UP","details":{"total":151051448320,"free":96219418624,"threshold":10485760,"exists":true}},"livenessStateHealthIndicator":{"status":"UP"},"dbHealthContributor":{"status":"UP","details":{"database":"PostgreSQL","validationQuery":"isValid()"}},"ebsHealthIndicator":{"status":"UNKNOWN","details":{"reason":"Timeout","thread":"main-pool-34","elapsed":"PT9.794250824S"}}}}

  • The UI displays: 'Failed to publish event to topic: Deployment requested'

Environment

Aria Automation 8.18.x

Cause

Quorum queues created by the Event Broker Service (EBS) in RabbitMQ do not automatically replicate across newly added nodes during a scale-out or node replacement. If the original node (where the queues were mastered) becomes unavailable, the system cannot publish or consume events.

Resolution

To replicate the queues across the RabbitMQ cluster executing the following steps:

• Ensure that all VMware Aria Automation nodes are up and running.
• ssh to the first node in case of scale out or to one of the original nodes in case some node is replaced and execute the below command.

  kubectl -n prelude exec -it rabbitmq-ha-0 -- rabbitmq-queues grow "[e-mail]" "all" --vhost-pattern "/" --queue-pattern ".*"
  
  kubectl -n prelude exec -it rabbitmq-ha-0 -- rabbitmq-queues grow "[e-mail]" "all" --vhost-pattern "/" --queue-pattern ".*"
  
  kubectl -n prelude exec -it rabbitmq-ha-0 -- rabbitmq-queues grow "[e-mail]" "all" --vhost-pattern "/" --queue-pattern ".*"

• These commands will ensure that each queue has replica on every node.

Additional Information

More information about rabbitmq-queues grow can be found here - https://www.rabbitmq.com/docs/man/rabbitmq-queues.8#grow