VMware Tanzu Service Broker for AWS broker won't start after upgrading to TAS v2.10.31+/ v2.11.19+/ v2.12.12+

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

During the upgrade to TAS v2.10.31/ v2.11.19+/ v2.12.12+, the broker fails on start up with error:

          FAILED  
          Error restarting application: Start unsuccessful  

From broker app log, it throws below error:

2022-04-26T21:12:01.26+1000 [APP/PROC/WEB/0] OUT {"timestamp":"1650971521.267858744","source":"aws-broker","message":"aws-broker.store failed to initialize","log_level":3,"data":{"error":"tls: server selected unsupported protocol version 301","trace":"goroutine 1 [running]:\ncode.cloudfoundry.org/lager.(*logger).Fatal(0xc0000b1da0, {0x55ed0b2b7218, 0x1a}, {0x55ed0b7051d8?, 0xc0004087c0}, {0x0, 0x0, 0x55ed0add6fa5?})\n\t/tmp/app/vendor/code.cloudfoundry.org/lager/logger.go:131 +0x1fc\nmain.main()\n\t/tmp/app/main.go:77 +0x66c\n"}}
2022-04-26T21:12:01.27+1000 [APP/PROC/WEB/0] ERR panic: tls: server selected unsupported protocol version 301
2022-04-26T21:12:01.27+1000 [APP/PROC/WEB/0] ERR goroutine 1 [running]:
2022-04-26T21:12:01.27+1000 [APP/PROC/WEB/0] ERR code.cloudfoundry.org/lager.(*logger).Fatal(0xc0000b1da0, {0x55ed0b2b7218, 0x1a}, {0x55ed0b7051d8?, 0xc0004087c0}, {0x0, 0x0, 0x55ed0add6fa5?})
2022-04-26T21:12:01.27+1000 [APP/PROC/WEB/0] ERR /tmp/app/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x4f0
2022-04-26T21:12:01.27+1000 [APP/PROC/WEB/0] ERR main.main()
2022-04-26T21:12:01.27+1000 [APP/PROC/WEB/0] ERR /tmp/app/main.go:77 +0x66c
2022-04-26T21:12:01.27+1000 [APP/PROC/WEB/0] OUT Exit status 2
2022-04-26T21:12:01.27+1000 [CELL/SSHD/0] OUT Exit status 0

Resolution

The error in the broker app implies that the RDS server is choosing the TLS version. Protocol version 301 = TLS 1.0 - that is insecure TLS version. The service broker app is built with the golang buildpack. The upgrade to TAS v2.10.31+/ v2.11.19+/ v2.12.12+ will introduce the change of golang buildpack to upgrade to v1.9.42 as per the release note here https://docs.pivotal.io/application-service/2-11/release-notes/runtime-rn.html#2.11.19. And from golang buildpack release page: https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.9.42. It added go 1.18 for stack(s) cflinuxfs3. TLS 1.0/1.1 is disabled by default in golang 1.18: https://tip.golang.org/doc/go1.18#tls10. Thus the root cause is that the RDS instance you're using as the broker DB doesn’t support TLS 1.2 yet. It needs to be updated to support golang 1.18 where TLS 1.1/1.0 have been removed for the permanent fix.

If you are unable to update your RDS instance immediately. The following workaround will force the broker to build with go 1.17:

cf target -o system -s iaas-brokers
cf set-env aws-services-broker GOVERSION go1.17
cf restage aws-services-broker

You should see a line like this in the staging logs:

-----> Installing go 1.17.8

Make sure to remove the workaround someday, at a minimum before golang 1.17 is removed from support:

cf unset-env aws-services-broker GOVERSION