Instance creation is stuck at scheduling

Products

• VMware Integrated OpenStack

Issue/Introduction

• Instances are not being created
• compute logs have messages similar to this:
  nova-compute.log:Jul 05 11:01:20 controller-r727nwvjfd nova-compute[768]: 2024-07-05 11:01:20.712 1 ERROR oslo_messaging._drivers.amqpdriver [req-42865880-d802-4dff-a864-a8bfe51e7339 cfb6ba0a723d4b12b9e192bdc98329e7 cb0df80270fe4525b0872e63be300e33 - default default] The reply 406972d7dcb24fbb95612edf9e0adf61 failed to send after 60 seconds due to a missing queue (reply_2c5433018dc34d638a326402baea50f6). Abandoning...: oslo_messaging.exceptions.MessageUndeliverable
  
  
• rabbitmq logs have messages similar to this:
  rabbitmq.log:Jul 02 20:56:10 controller-4fm5zkgvrc rabbitmq[757]:  operation queue.declare caused a connection exception internal_error: "Cannot declare a queue 'queue 'reply_b3f355e4367d4a4e865b3bd7e5b1b817' in vhost 'nova'' on node '[email protected]': nodedown"
rabbitmq.log:Jul 02 20:56:11 controller-4fm5zkgvrc rabbitmq[757]: 2024-07-02 20:56:11.259 [info] <0.12004.7805> Mirrored queue 'notifications.error' in vhost 'glance': Slave <[email protected]> saw deaths of mirrors 

Note:  The preceding log excerpts are only examples.  Date, time and environmental variables may vary depending on your environment.

Environment

7.3

Cause

The rabbitmq service/cluster has a problem.

Resolution

1. Restart the rabbitmq service/cluster.
   # kubectl rollout restart statefulset rabbitmq1-rabbitmq -n openstack
2. Watch for the rabbitmq pods to restart.
   # pods
# viocli get deployment
3. Once these pods restart, retry your operation.