Spring Cloud Service (SCS) for VMware Tanzu 2.x upgrade fails at Deploy Broker Errands

Products

• Support Only for Spring

Issue/Introduction

When upgrading the Spring Cloud Services (SCS) for for VMware Tanzu 2.x tile, Apply Changes fails at Deploy Broker Errands.


Scenario 1

In Operations Manager's Apply Changes log, you see the following error message:

Starting to wait for venerable broker to become idle 

Failed to fetch https://spring-cloud-broker-venerable.<DOMAIN>/idle_time: status code 404. Continuing to wait for venerable broker to become idle 

Failed to fetch https://spring-cloud-broker-venerable.<DOMAIN>/idle_time: status code 404. Continuing to wait for venerable broker to become idle 

Failed to fetch https://spring-cloud-broker-venerable.<DOMAIN>/idle_time: status code 404. Continuing to wait for venerable broker to become idle 

Failed to fetch https://spring-cloud-broker-venerable.<DOMAIN>/idle_time: status code 404. Continuing to wait for venerable broker to become idle 

Failed to fetch https://spring-cloud-broker-venerable.<DOMAIN>/idle_time: status code 404. Continuing to wait for venerable broker to become idle 

Failed to fetch https://spring-cloud-broker-venerable.<DOMAIN>/idle_time: status code 404. Continuing to wait for venerable broker to become idle 

Failed to fetch https://spring-cloud-broker-venerable.<DOMAIN>/idle_time: status code 404. Continuing to wait for venerable broker to become idle 

Scenario 2

In addition to the Apply changes logs shown above,  you see these errors in the output of deploy-service-broker errand and  in the  logs of spring-cloud-broker-venerable
 

Output of deploy-service-broker errand  logs

cf env spring-cloud-broker-venerable
Starting to wait for venerable broker to become idle
Venerable broker has been idle for 368 seconds
Operation eventually succeeded
cf delete spring-cloud-broker-venerable -f -r
Deleting app spring-cloud-broker-venerable in org system / space p-spring-cloud-services as admin...
FAILED
Server error, status code: 504, error code: 10001, message: An unbind operation for the service binding between app spring-cloud-broker-venerable and service instance spring-cloud-broker-db failed: The request to the service broker timed out: http://192.168.10.4:8080/v2/service_instances/52946a61-304a-4d0a-97b0-45e36149420c/service_bindings/b8ebe6d8-5a06-4190-87bf-aecab569fb5a?accepts_incomplete=true&plan_id=07cf3296-b8e3-4c6b-8d74-b11a89599627&service_id=548966e5-e333-4d65-8773-7b4e3bb6ca97
deploy-service-broker failed: exit status 1


Retrieving logs for app spring-cloud-broker-venerable in org system / space p-spring-cloud-services as 685981698...

   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at org.springframework.boot.actuate.autoconfigure.endpoint.web.ServletEndpointManagementContextConfiguration$WebMvcServletEndpointManagementContextConfiguration$$EnhancerBySpringCGLIB$$69d54e81$$FastClassBySpringCGLIB$$587db901.invoke(<generated>) ~[spring-boot-actuator-autoconfigure-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:365) ~[spring-context-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at org.springframework.boot.actuate.autoconfigure.endpoint.web.ServletEndpointManagementContextConfiguration$WebMvcServletEndpointManagementContextConfiguration$$EnhancerBySpringCGLIB$$69d54e81.servletEndpointRegistrar(<generated>) ~[spring-boot-actuator-autoconfigure-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  ... 24 common frames omitted
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthEndpoint]: Factory method 'healthEndpoint' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration$$EnhancerBySpringCGLIB$$2285482c]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'spring-cloud-broker-db': Post-processing of FactoryBean's singleton object failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker': Invocation of init method failed; nested exception is org.springframework.jdbc.datasource.init.UncategorizedScriptException: Failed to execute database script; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLInvalidAuthorizationSpecException: Access denied for user 'b8ebe6d85a06419087bfaecab569fb5a'@'10.12.0.4' (using password: YES)
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:583) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
   2021-11-04T13:04:55.88-0400 [APP/PROC/WEB/0] OUT  ... 48 common frames omitted

Resolution

For Scenario 1 described above, 

Please go to org system space p-spring-cloud-services and delete the spring-cloud-broker-venerable and spring-cloud-broker-worker-venerable apps.

Once these two apps are deleted, please rerun the Apply Changes on SCS and Apply Changes should now succeed. 

For Scenario2 described above, 

Please go to org system space p-spring-cloud-services and delete spring-cloud-broker, spring-cloud-broker-worker, spring-cloud-broker-venerable and spring-cloud-broker-worker-venerable apps.

Once all four apps are deleted, please rerun the Apply Changes on SCS and Apply Changes should now succeed.