Cloud Service Broker (CSB) for Azure upgrade to 1.11 fails
Article ID: 389016
Updated On:
Products
Issue/Introduction
While upgrading CSB for Azure from 1.10 to 1.11, they saw an error like this in the Ops Manager change log:
Errand 'register-broker' completed with error (exit code 1)
Exit code 1
Task 1628757 Duration 00:00:05
Task 1628757 done
Instance cloud-service-broker/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Exit Code 1
Stdout cf version 8.8.3+3cd802e.2024-10-29
Setting API endpoint to api.system-domain.example.com...
OK
API endpoint: https://api.system-domain.example.com
API version: 3.178.0
Not logged in. Use 'cf login' or 'cf login --sso' to log in.
API endpoint: https://api.system-domain.example.com
Authenticating...
OK
Use 'cf target' to view or set your target org and space.
Updating service broker cloud-service-broker-azure as system_services...
FAILED
Stderr Job (14bf37a6-8515-43d5-b6e0-74e5d5d5db50) failed: The service broker returned an invalid response. Status Code: 502 Bad Gateway, Body: 502 Bad Gateway: Registered endpoint failed to handle the request.
CSB deployment logs were pulled using bosh (bosh -d <csb-deployment> logs) and the following errors appeared in the csb-app.stdout.log file:
2025/02/20 20:11:20 [35m/var/vcap/data/compile/csb/[email protected]/dbservice/setup_db.go:143
[0m[31m[error] [0mfailed to initialize database, got error dial tcp XXX.XXX.XXX.XXX:3306: i/o timeout
{"timestamp":"1740082280.907014608","source":"cloud-service-broker","message":"cloud-service-broker.Database Setup","log_level":2,"data":{"error":"dial tcp XXX.XXX.XXX.XXX:3306: i/o timeout"}}
{"timestamp":"1740082282.028143883","source":"cloud-service-broker","message":"cloud-service-broker.starting","log_level":1,"data":{"version":"0.0.0"}}
{"timestamp":"1740082282.028263330","source":"cloud-service-broker","message":"cloud-service-broker.Connecting to MySQL Database","log_level":1,"data":{"host":"db-hostname.example.com","name":"servicebroker","port":"3306","tls":"\u0026tls=true","username":"[email protected]"}}Cause
Tanzu Cloud Service Broker for Azure v1.11 changed its architecture. The broker no longer runs as an app on Tanzu Application Service; it deploys its own bosh-deployed VMs and runs the csb-azure app there. Consequently, a firewall rule must be enabled to connect from the broker's network allowing connection to the IP address and port of the MySQL state database configured in the CSB tile.
Resolution
- Make sure there is a firewall rule allowing connection to the IP address and port of the MySQL state database configured in the CSB tile.
- Make sure the state database is running as expected.
- Make sure the IP address and port are correctly configured in the tile.
To test whether the needed firewall rule is present, bosh ssh to one of the cloud service broker VMs. Try to connect to the state database at the IP address and port configured in the tile with this command:
nc -vz IP_ADDRESS PORT
If the command succeeds, the rule is present. If the command hangs, then the rule is missing.
Additional Information
Cloud Service Broker for AWS v1.13 also switched from an app deployed on TAS to bosh-deployed VMs. As a result, it also requires a firewall rule to be enabled in order to allow connections from the VMs to the backing MySQL database.
Feedback
