BOSH upload-release times out after 15 minutes on random releases
Article ID: 293572
Updated On:
Products
Operations Manager
Issue/Introduction
Symptoms:
You are unable to run a deployment. The deployment fails during the upload-releases phase.
This would seem to be a random failure, as it depends on the load on the Director database instance. This issue is observed when an external database instance is used, primarily AWS but this could affect GCP or Azure too, because the limited IOPS on the external instance can stall the upstream tasks from the Director VM.
The upload-releases task times out after 15min and retries for 5 times before bailing out of the deployment.
The error seen in the deployment is below:
You are unable to run a deployment. The deployment fails during the upload-releases phase.
This would seem to be a random failure, as it depends on the load on the Director database instance. This issue is observed when an external database instance is used, primarily AWS but this could affect GCP or Azure too, because the limited IOPS on the external instance can stall the upstream tasks from the Director VM.
The upload-releases task times out after 15min and retries for 5 times before bailing out of the deployment.
The error seen in the deployment is below:
===== 2018-08-30 16:17:50 UTC Running "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=192.168.1.11 upload-release /var/tempest/releases/php-offline-buildpack-4.3.57-3586.27.0.tgz" Using environment '192.168.1.11' as client 'ops_manager' 100.00% 0 B/s 15m0s Uploading release file: Director responded with non-successful status code '504' response '<html> <head><title>504 Gateway Time-out</title></head> <body bgcolor="white"> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx</center> </body> </html>
Environment
Cause
The external database instance used for BOSH Director needs at least 1500 IOPS.
AWS assigns finite number of IOPS depending on the size of the Allocated Storage.
A smaller storage, like 20GB in this case, has less IOPS (around 600 IOPS in this case).
This caused the database instance to stall while the upload-releases was running.
For AWS, please see the details in the docs for AWS RDS Database configuration for details.
AWS assigns finite number of IOPS depending on the size of the Allocated Storage.
A smaller storage, like 20GB in this case, has less IOPS (around 600 IOPS in this case).
This caused the database instance to stall while the upload-releases was running.
For AWS, please see the details in the docs for AWS RDS Database configuration for details.
Resolution
Allocated Storage should be selected as per the IOPS requirements.
For AWS, Increase the Allocated Storage to at least 500GB to get enough IOPS to meet minimum requirements. See Additional information below for details on why 500GB is required.
For AWS, Increase the Allocated Storage to at least 500GB to get enough IOPS to meet minimum requirements. See Additional information below for details on why 500GB is required.
Feedback
Yes
No
Powered by
