After upgrade from TAS 2.11.15 to 2.11.17 connecting to RabbitMQ using TCP router intermittently fails

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

After the upgrade, when connecting to a RabbitMQ service instance with "Service-Gateway Access" from a computer outside CF, this fails intermittently.
When watching the output of the PowerShell cmdlet command :

Test-NetConnection example-rabbitmq.sys.nonprod02.ap.example.com -Port 1140

The connection is successful for 40 seconds and then for 20 seconds not. When the normal time ends with 00-20 seconds, no connection can be made and from 21-59 seconds a connection can be made.
RabbitMQ instance with bosh:

rabbitmq-server/e90ecaf1-77f7-4e70-8e52-9b1e86c773dc:/var/vcap/sys/log/route_registrar#less route_registrar.stdout.log

{"timestamp":"1649247109.854146004","source":"Route Registrar","message":"Route Registrar.Registered routes successfully","log_level":1,"data":{}}
{"timestamp":"1649247129.831651211","source":"Route Registrar","message":"Route Registrar.no healthchecker found for route","log_level":1,"data":{"route":{"Type":"","Name":"rmq-21df53a5-7850-4ae8-b5c3-92be945477a1","Port":null,"TLSPort":15671,"Tags":null,"URIs":["rmq-21df53a5-7850-4ae8-b5c3-92be945477a1.sys.nonprod02.ap.example.com"],"RouterGroup":"","Host":"","ExternalPort":null,"RouteServiceUrl":"","RegistrationInterval":20000000000,"HealthCheck":null,"ServerCertDomainSAN":"rmq-21df53a5-7850-4ae8-b5c3-92be945477a1.sys.nonprod02.ap.example.com"}}}
{"timestamp":"1649247129.831750870","source":"Route Registrar","message":"Route Registrar.Registering route","log_level":1,"data":{"route":{"Type":"","Name":"rmq-21df53a5-7850-4ae8-b5c3-92be945477a1","Port":null,"TLSPort":15671,"Tags":null,"URIs":["rmq-21df53a5-7850-4ae8-b5c3-92be945477a1.sys.nonprod02.ap.example.com"],"RouterGroup":"","Host":"","ExternalPort":null,"RouteServiceUrl":"","RegistrationInterval":20000000000,"HealthCheck":null,"ServerCertDomainSAN":"rmq-21df53a5-7850-4ae8-b5c3-92be945477a1.sys.nonprod02.ap.example.com"}}}
{"timestamp":"1649247129.831773520","source":"Route Registrar","message":"Route Registrar.Registered routes successfully","log_level":1,"data":{}}
{"timestamp":"1649247129.831789255","source":"Route Registrar","message":"Route Registrar.no healthchecker found for route","log_level":1,"data":{"route":{"Type":"tcp","Name":"RMQ-service-instance_21df53a5-7850-4ae8-b5c3-92be945477a1-AMQP","Port":5671,"TLSPort":null,"Tags":null,"URIs":null,"RouterGroup":"RMQ-service-instance_21df53a5-7850-4ae8-b5c3-92be945477a1-AMQP","Host":"10.4.248.43","ExternalPort":1140,"RouteServiceUrl":"","RegistrationInterval":20000000000,"HealthCheck":null,"ServerCertDomainSAN":""}}}
{"timestamp":"1649247129.831822634","source":"Route Registrar","message":"Route Registrar.Registering route","log_level":1,"data":{"route":{"Type":"tcp","Name":"RMQ-service-instance_21df53a5-7850-4ae8-b5c3-92be945477a1-AMQP","Port":5671,"TLSPort":null,"Tags":null,"URIs":null,"RouterGroup":"RMQ-service-instance_21df53a5-7850-4ae8-b5c3-92be945477a1-AMQP","Host":"10.4.248.43","ExternalPort":1140,"RouteServiceUrl":"","RegistrationInterval":20000000000,"HealthCheck":null,"ServerCertDomainSAN":""}}}
{"timestamp":"1649247129.831840038","source":"Route Registrar","message":"Route Registrar.Creating mapping","log_level":1,"data":{}}
{"timestamp":"1649247129.838819504","source":"Route Registrar","message":"Route Registrar.Upserted route","log_level":1,"data":{"route-mapping":{"router_group_guid":"9f1b03dd-9477-4190-4510-0e56ccc7cff4","backend_port":5671,"backend_ip":"10.4.248.43","port":1140,"modification_tag":{"guid":"","index":0},"ttl":20,"isolation_segment":""}}}
{"timestamp":"1649247129.838853121","source":"Route Registrar","message":"Route Registrar.Registered routes successfully","log_level":1,"data":{}} 

# ssh to tcp_router tcp_router/b94242a5-3464-4632-a132-564f40f5713a:/var/vcap/sys/log/tcp_router

watch "cat /var/vcap/data/tcp_router/config/haproxy.conf | grep 1140"

This gives the route for 1140 for some time, but then this part in the haproxy.conf file disappears.

Environment

Product Version: 2.11

Resolution

This issue happen because of race condition that is happening when 2 routers are in different version.

TAS 2.11.15
     Routing: 0.229.0
TAS 2.11.17+
     Routing: 0.231.0
RabbitMQ < 2.0.12
     Routing: 0.229.0
RabbitMQ 2.0.12+
     Routing: 0.231.0

RabbitMQ need to be upgraded to 2.0.12+ to match the version of router service in the TAS.