Apply Changes fails when updating first pre-provisioned RabbitMQ server due to post-start script failure using custom config that uses "auth_backends"

Products

• VMware RabbitMQ

Issue/Introduction

If using a custom configuration that uses "auth_backends", you may experience an issue like the following when applying changes on the pre-provisioned servers:

Task 120769

Task 120769 | 19:56:27 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 120769 | 19:56:27 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 120769 | 19:56:27 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 120769 | 19:56:28 | Preparing deployment: Preparing deployment (00:00:03)
Task 120769 | 19:56:31 | Preparing deployment: Rendering templates (00:00:06)
Task 120769 | 19:56:37 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 120769 | 19:56:38 | Updating instance rabbitmq-server: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:56:40 | L executing pre-stop: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:56:41 | L executing drain: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:56:50 | L stopping jobs: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:56:58 | L executing post-stop: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:57:16 | L installing packages: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:57:18 | L configuring jobs: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:57:18 | L executing pre-start: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:57:26 | L starting jobs: rabbitmq-server/example1 (0) (canary)
Task 120769 | 19:57:56 | L executing post-start: rabbitmq-server/example1 (0) (canary) (00:15:40)
Updating deployment:
 Expected task '120769' to succeed but state is 'error'
Exit code 1
                     L Error: Action Failed get_task: Task 8531c082-cc7e-4cfb-554f-a1d564e14fe3 result: 1 of 2 post-start scripts failed. Failed Jobs: rabbitmq-server. Successful Jobs: bosh-dns.
Task 120769 | 20:12:18 | Error: Action Failed get_task: Task 8531c082-cc7e-4cfb-554f-a1d564e14fe3 result: 1 of 2 post-start scripts failed. Failed Jobs: rabbitmq-server. Successful Jobs: bosh-dns.

Task 120769 Started  Thu Jun 10 19:56:27 UTC 2021
Task 120769 Finished Thu Jun 10 20:12:18 UTC 2021
Task 120769 Duration 00:15:51
Task 120769 error

Further investigation of the post-start script yields the following:

2021-06-10T20:12:15Z: Running cluster checks from post-start...
Testing TCP connections to all active listeners on node rabbit@example ...
Successfully connected to ports 5672, 15672, 15692, 25672 on node rabbit@example.
Checking if all vhosts are running on node rabbit@example ...
Node rabbit@example reported all vhosts as running
Authenticating user "broker" ...
Error:
Error: failed to authenticate user "broker"
[email protected]
User 'broker' cannot authenticate
2021-06-10T20:12:17Z: RabbitMQ cluster is not healthy

Resolution

When using custom config with pre-provisioned servers, you have to make sure that "rabbit_auth_backend_internal" is set in the "auth_backends" key like so:

{auth_backends, [rabbit_auth_backend_cache, rabbit_auth_backend_ldap, rabbit_auth_backend_internal]}

This is because the broker user is used to create users and vhosts in the cluster and must be able to authenticate. Once you add the internal backend, then the deployment should go through with no issue.