ERROR: Resource DELETE failed: MessagingTimeout: resources.evan-cluster: Timed out waiting for a reply to message ID <message UUID>
search cancel

ERROR: Resource DELETE failed: MessagingTimeout: resources.evan-cluster: Timed out waiting for a reply to message ID <message UUID>

book

Article ID: 396292

calendar_today

Updated On: 05-02-2025

Products

VMware Integrated OpenStack

Issue/Introduction

  • Getting errors when creating or deleting stack
  • Running the this command will show an error similar to the following:
    $ openstack stack show <stack name/id>
    | stack_status          | DELETE_FAILED  
    | stack_status_reason   | Resource DELETE failed: MessagingTimeout: resources.evan-cluster: Timed out waiting for a reply to message ID <message UUID>

Environment

7.x

Cause

This can be caused by an issue with the RabbitMQ cluster or the MariaDB cluster.

Resolution

  1. Check for any RabbitMQ or MariaDB pods in an error status
    #osctl get pods | grep -ie 'mariadb|rabbitmq'
  2. If there any in error, determine the cause of the error.  
  3. Check the MariaDB logs to see if we have errors similar (service user and host may change) repeating in the logs of any one node:
    [Warning] Aborted connection 9444116 to db: '<openstack service>' user: '<openstack service>' host: '<host IP>' (Got an error reading communication packets)"


  4. Determine which MariaDB node this is and restart that pod:
    #osdel pod mariadb-server-<1, 2 or 3 depending on which node the errors were on>
  5. Wait for pod to restart.
  6. Once the pod is restarted, wait a few minutes and retry your stack operation.

Additional Information

We can restart the RabbitMQ cluster using the following command:

#kubectl rollout restart statefulset rabbitmq1-rabbitmq -n openstack