Running Spring Cloud Services upgrade-all-instances fails with 'Unknown Cloud Foundry Exception'

Products

• Support Only for Spring

Issue/Introduction

It can happen that running Spring Cloud Services upgrade-all-instances fails with
 

   org.cloudfoundry.UnknownCloudFoundryException: Unknown Cloud Foundry Exception  
Exit code 1
           	at org.cloudfoundry.reactor.util.ErrorPayloadMappers.lambda$null$11(ErrorPayloadMappers.java:114) ~[cloudfoundry-client-reactor-5.7.0.RELEASE.jar!/:na]  
           	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:   
           Assembly trace from producer [reactor.core.publisher.MonoFlatMap] :  
           	reactor.core.publisher.Mono.checkpoint(Mono.java:2177)  
           	org.cloudfoundry.reactor.client.v2.serviceinstances.ReactorServiceInstances.get(ReactorServiceInstances.java:88)  
           Error has been observed at the following site(s):  
           	*__checkpoint() ⇢ at org.cloudfoundry.reactor.client.v2.serviceinstances.ReactorServiceInstances.get(ReactorServiceInstances.java:88)  
           Original Stack Trace:  
           		at org.cloudfoundry.reactor.util.ErrorPayloadMappers.lambda$null$11(ErrorPayloadMappers.java:114) ~[cloudfoundry-client-reactor-5.7.0.RELEASE.jar!/:na]  
           		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) ~[reactor-core-3.4.18.jar!/:3.4.18]  
           		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.4.18.jar!/:3.4.18]  
           		at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:126) ~[reactor-core-3.4.18.jar!/:3.4.18]  

And when checking the previous lines, you see that one ore more services instances failed with "is in 'failed' state. It cannot be upgraded." as below.
 

2023-09-26 12:57:26.677  WARN 6 --- [ctor-http-nio-1] i.p.s.c.s.e.core.CloudFoundryService     : Service instance with id 70d649a2-b258-481f-9dae-8fff5448cdf8 is in 'failed' state. It cannot be upgraded. 

Environment

Product Version: 3.1

Resolution

Locating those service instance backing apps from the instance id in logs lines, as explained here, and restarting them will make the errand run successfully next time.