Cloud Service Broker for AWS upgrade-services errand fails with "error upgrade request timeout"
Article ID: 418015
Updated On:
Products
Issue/Introduction
The upgrade-services errand of Cloud service Broker for AWS might fail to update some existing service instances and reports errors like the following.
===== 2025-11-12 02:19:55 UTC Running "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=##.##.##.## --deployment=cloud-service-broker-aws-#### run-errand upgrade-services --instance cloud-service-broker/first"
Using environment '##.##.##.##' as client 'ops_manager'
......
2025-11-12T02:19:57Z: starting to upgrade instance: "push-notification-service-database" guid: "10e798b6-####-747fa4bc1359"
2025-11-12T02:29:59Z: upgrade of instance: "push-notification-service-database" guid: "10e798b6-####-747fa4bc1359" failed after 10m1.979945819s: error upgrade request timeout
......
Details: "error upgrade request timeout"
Service Instance Name: "push-notification-service-database"
Service Instance GUID: "10e798b6-####-747fa4bc1359"
Service Instance Version: "1.9.1"
Service Plan Name: "large"
Service Plan GUID: "835701b4-####-2a426e947f37"
Service Plan Version: "1.10.6"
Service Offering Name: "csb-aws-postgresql"
Service Offering GUID: "1a058acc-####-7c5563ac1218"
Space Name: "SP1"
Space GUID: "f0069181-####-018a154682ef"
Organization Name: "ORG1"
Organization GUID: "03a30ba0-####-f2c222da2c9c"
Instance upgrade failed. Review the logs for issues during instance upgrade.
Stderr upgrade-all-services plugin error: there were failures upgrading one or more instances. Review the logs for more information
Environment
- Cloud Service Broker for AWS
Cause
There is a 10-minute timer in broker app for updating every single service instance. If the update process couldn't complete within 10 minutes, the "error upgrade request timeout" will be reported by upgrade-services errand.
Resolution
After submitting the update request for an existing service instance, the update operation is actually performed asynchronously on the platform. It's possible for some large service instance to take more than 10 minutes to complete the update.
So when "error upgrade request timeout" is reported for some service instance, the update process could be still going on in the background.
The default 10-minute timer is not configurable at the moment.
You can try to check the update progress with "/v3/service_instances/<guid>" CAPI. For example,
$ cf curl /v3/service_instances/10e798b6-####-747fa4bc1359 | jq .
{
"guid": "10e798b6-####-747fa4bc1359",
"created_at": "2024-04-02T23:14:26Z",
"updated_at": "2025-11-12T02:30:05Z",
"name": "push-notification-service-database",
"tags": [],
"last_operation": {
"type": "update",
"state": "succeeded",
"description": "upgrade succeeded: created db db1 (id: db-XDAT####DC3QQ) on server csb-postgresql-10e798b6-####-747fa4bc1359.####.ap-southeast-2.rds.example.com URL: https://ap-southeast-2.console.aws.example.com/rds/home?region=ap-southeast-2#database:id=XDAT####DC3QQ;is-cluster=false",
"updated_at": "2025-11-12T02:30:05Z",
"created_at": "2025-11-12T02:30:05Z"
}Different actions could be taken according to the state value in the last_operation section.
| State | Action |
|
succeeded |
Try to run the upgrade-services errand again if all problematic service instances show succeeded state. The errand should complete quickly without any error. If not, contact Tanzu support |
|
in progress |
Keep checking the state till it's changed to succeeded or failed. If it's abnormally stuck in this state for a very long time, contact Tanzu support |
|
failed |
Review the broker logs for errors related to the service instance. Contact Tanzu support if needed. |
Feedback
