RabbitMQ Server IP address changes following update to the Service Instance.
Article ID: 298438
Updated On:
Products
VMware Tanzu Application Service for VMs
Issue/Introduction
This Knowledge Base (KB) article details a behavior in RabbitMQ for Tanzu Application Service starting since tile version 1.20.
The specific behavior is the IP address of the RabbitMQ server(s) changing during an update to the Service Instance (SI) deployment.
As an example, consider the following output from BOSH cli:
Notice how the IPs of the RabbitMQ servers changed from (10.3.3.24, 10.3.3.25, 10.3.3.26) to (10.3.3.27, 10.3.3.28, 10.3.3.29).
The reason for this is derived from a BOSH update strategy called create-swap-delete. Starting since RabbitMQ for Tanzu Application Service tile version 1.20 the service instance deployments are assigned this strategy. It can be confirmed in the SI deployment manifest:
The specific behavior is the IP address of the RabbitMQ server(s) changing during an update to the Service Instance (SI) deployment.
As an example, consider the following output from BOSH cli:
# Before the update of deployment ubuntu@opsman:~$ bosh -d service-instance_bc3ee798-6fc0-4eb1-a091-7b133a3ddc71 vms Using environment '10.1.1.2' as client 'ops_manager' Task 804. Done Deployment 'service-instance_bc3ee798-6fc0-4eb1-a091-7b133a3ddc71' Instance Process State AZ IPs VM CID VM Type Active Stemcell rabbitmq-server/789ce958-c07b-4950-bc9e-f02b3c1d3e83 running az1 10.3.3.25 rabbitmq-server_service-instance-bc3ee798-6fc0-4eb1-a091-7b133a3dd_75b9d403dc25 large true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.340 rabbitmq-server/c10c02dd-a596-4e1d-b087-1c6d4e6efc52 running az1 10.3.3.26 rabbitmq-server_service-instance-bc3ee798-6fc0-4eb1-a091-7b133a3dd_3b5bc2c7c9a1 large true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.340 rabbitmq-server/f8dba482-fb85-44d0-9a88-ef9b9c138c7d running az1 10.3.3.24 rabbitmq-server_service-instance-bc3ee798-6fc0-4eb1-a091-7b133a3dd_2bda73f9f4b8 large true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.340
# After the update of deployment ubuntu@opsman:~$ bosh -d service-instance_bc3ee798-6fc0-4eb1-a091-7b133a3ddc71 vms Using environment '10.1.1.2' as client 'ops_manager' Task 888. Done Deployment 'service-instance_bc3ee798-6fc0-4eb1-a091-7b133a3ddc71' Instance Process State AZ IPs VM CID VM Type Active Stemcell rabbitmq-server/789ce958-c07b-4950-bc9e-f02b3c1d3e83 running az1 10.3.3.28 rabbitmq-server_service-instance-bc3ee798-6fc0-4eb1-a091-7b133a3dd_fd590e856520 large true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.351 rabbitmq-server/c10c02dd-a596-4e1d-b087-1c6d4e6efc52 running az1 10.3.3.29 rabbitmq-server_service-instance-bc3ee798-6fc0-4eb1-a091-7b133a3dd_89bd4b1891cf large true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.351 rabbitmq-server/f8dba482-fb85-44d0-9a88-ef9b9c138c7d running az1 10.3.3.27 rabbitmq-server_service-instance-bc3ee798-6fc0-4eb1-a091-7b133a3dd_219820c392dd large true bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.351 3 vms Succeeded
Notice how the IPs of the RabbitMQ servers changed from (10.3.3.24, 10.3.3.25, 10.3.3.26) to (10.3.3.27, 10.3.3.28, 10.3.3.29).
The reason for this is derived from a BOSH update strategy called create-swap-delete. Starting since RabbitMQ for Tanzu Application Service tile version 1.20 the service instance deployments are assigned this strategy. It can be confirmed in the SI deployment manifest:
ubuntu@opsman:~$ bosh -d service-instance_bc3ee798-6fc0-4eb1-a091-7b133a3ddc71 manifest | grep strategy vm_strategy: create-swap-delete
Environment
Product Version: Other
Resolution
The broker deployment manifest will show that it has hot swap enabled:
Rare Issue
In the event that your environment has legacy Service Instance bindings or legacy service keys created for a RabbitMQ SI prior to TAS 2.x, then the ampq connection string may reference the IP of the RabbitMQ server instead of the hostname. For example it may look like:
The legacy connection string will be invalid for the SIs that have the create-swap-delete vm_strategy as the IP address of the RabbitMQ server will change during the deployment update. For this scenario the following recommendations should resolve:
ubuntu@opsman:~$ bosh -d p-rabbitmq-d6ce34cad8ad1845ef8e manifest | grep swap create_swap_delete: true
- Currently this property is not exposed as configurable and defaults to true starting since tile version 1.20.
- Hot swap is disabled for all SIs created prior to tile version 1.20 and cannot be enabled.
- Hot swap is enabled for all SIs created after tile version 1.20 and cannot be disabled.
Rare Issue
In the event that your environment has legacy Service Instance bindings or legacy service keys created for a RabbitMQ SI prior to TAS 2.x, then the ampq connection string may reference the IP of the RabbitMQ server instead of the hostname. For example it may look like:
# Connection String for SI created before TAS 2.x amqp://e63b322b-ca4e-4860-a6df-580f7ca6fc51:[REDACTED]@10.3.3.25/bc3ee798-6fc0-4eb1-a091-7b133a3ddc71 # Connection String for SI created after TAS 2.x amqp://e63b322b-ca4e-4860-a6df-580f7ca6fc51:[REDACTED]@q-s0.rabbitmq-server.service.service-instance-bc3ee798-6fc0-4eb1-a091-7b133a3ddc71.bosh/bc3ee798-6fc0-4eb1-a091-7b133a3ddc71
The legacy connection string will be invalid for the SIs that have the create-swap-delete vm_strategy as the IP address of the RabbitMQ server will change during the deployment update. For this scenario the following recommendations should resolve:
- If the app is consuming the connection string from VCAP_SERVICES then unbinding and rebinding the application to the SI should allow it to regenerate new credentials and the new connection string will reference the bosh url of the RabbitMQ server(s) instead of the IP address.
- If the app is consuming the connection string from service-key then creating a new service-key for the SI should generate a new connection string that contains the bosh url of the RabbitMQ server(s) instead of the IP address. Give this connection string to the application.
Feedback
Yes
No
Powered by
