Upgrade of RabbitMQ/REDIS/MYSQL fails with error: failed to create uaa client
Article ID: 293248
Updated On:
Products
VMware RabbitMQ
Issue/Introduction
Errors observed in the on-demand service broker:
To further confirm the problem from [on-demand-service-broker] run:
Note: Run them without -k option.
[on-demand-service-broker] [ID] <TIMESTAMP> error: failed to create uaa client: An error occurred while calling https://uaa.<SYSTEMDOMAIN>/oauth/clients. error for user: There was a problem completing your request. Please contact your operations team providing the following information: service: <SERVICE>, service-instance-guid: <SERVICEGUID>, broker-request-id: <ID> operation: create.
To further confirm the problem from [on-demand-service-broker] run:
Note: Run them without -k option.
curl -v https://uaa.<SYSTEMDOMAIN</info > external.out curl -v https://uaa.service.cf.internal:8443/info > internal.out
Inspect the output and confirm if response 200 is received or not.
The reason for this behavior is that with version of the broker 0.40, there is specific API call with strict tls handshake completed to uaa.service.cf.internal which, in case there is a load balancer, is resolved with LB IP and if the LB certificate is not present to the broker the request will fail.
To confirm this is the case, get the certificate used when connecting to UAA:
openssl s_client -showcerts -servername uaa.<SYSTEMDOMAIN> -connect uaa.<SYSTEMDOMAIN>:443 </dev/null
Then try with curl:
curl -v http://uaa.<SYSTEMDOMAIN>/info --cacert <CACERT from above command>
Environment
Product Version: 1.19
Resolution
https://github.com/pivotal-cf/on-demand-service-broker-release/releases/tag/v0.40.0
For a permanent solution to fix the Service broker, if you configure trusted certificates, it should work just fine and be more secure.
What you need to do:
Get the CA certificate used to sign the certificate on your external load balancer. Add that CA certificate to the BOSH trusted certificates (as documented in https://docs.pivotal.io/ops-manager/2-10/vsphere/config.html#security) and redeploy.
This way ODB broker should be able to establish the connection with UAA through the load balancer. The bug in ODB should be fixed in near future, but only affects customers who want to disable TLS validation which is obviously not recommended.
The workaround for the problem is to bypass the load balancer by directly defining in the /etc/hosts file the internal UAA IP address in the following format:
For a permanent solution to fix the Service broker, if you configure trusted certificates, it should work just fine and be more secure.
What you need to do:
Get the CA certificate used to sign the certificate on your external load balancer. Add that CA certificate to the BOSH trusted certificates (as documented in https://docs.pivotal.io/ops-manager/2-10/vsphere/config.html#security) and redeploy.
This way ODB broker should be able to establish the connection with UAA through the load balancer. The bug in ODB should be fixed in near future, but only affects customers who want to disable TLS validation which is obviously not recommended.
The workaround for the problem is to bypass the load balancer by directly defining in the /etc/hosts file the internal UAA IP address in the following format:
<UAA Server IP> uaa.<SYSTEMDOMAIN <UAA Server IP> uaa.service.cf.internal
Feedback
Yes
No
Powered by
