Large log bundle fails to download from BOSH
Article ID: 293726
Updated On:
Products
Operations Manager
Issue/Introduction
When an instance group log bundle is requested from BOSH that is larger than 5 GB, BOSH will give a 413 Request Entity Too Large error.
Here is an example of how it looks if trying to obtain the logs the command line:
Here is an example of how it looks when OpsManager fails to download:
Here is an example of how it looks if trying to obtain the logs the command line:
Here is an example of how it looks when OpsManager fails to download:
Environment
Product Version: 2.9
Resolution
When log bundles are requested from BOSH, the BOSH blobstore nginx server is involved. We can check the blobstore logs to get insight into recent transactions, such as log bundle requests.
To verify the reason a log bundle request failed:
1 - SSH onto the BOSH director and become root.
2 - change to directory /var/vcap/sys/log/blobstore
3 - tail -100 blobstore_error.log
When a log bundle is too large it will show errors similar to the following:
In order to change this value to larger than 5GB, perform the following:
1 - SSH onto the BOSH director and become root.
2 - Change to directory /var/vcap/jobs/blobstore/config/sites
3 - make a copy of the blobstore.conf
4 - Open the original blobstore.conf with an editor and increase the value of parameter "client_max_body_size" from 5000m to a larger size (in the above example we see the log bundle is 5.4GB so we would change this value to 6000m which is equal to 6GB) - save the changes.
5 - Restart the blobstore_nginx job with monit restart blobstore_nginx
When the blobstore_nginx job status is running with the new configuration, the log bundle request should succeed.
To verify the reason a log bundle request failed:
1 - SSH onto the BOSH director and become root.
2 - change to directory /var/vcap/sys/log/blobstore
3 - tail -100 blobstore_error.log
When a log bundle is too large it will show errors similar to the following:
2020/12/16 21:42:16 [error] 16#0: *1385 client intended to send too large body: 5408395472 bytes, client: 10.0.0.51, server: , request: "PUT /c4/66417b7a-08a3-441a-b884-f8ae5e59dd23 HTTP/1.1", host: "10.0.0.51:25250" 2020/12/16 21:42:16 [error] 13#0: *1386 client intended to send too large body: 5408395472 bytes, client: 10.0.0.51, server: , request: "PUT /c4/66417b7a-08a3-441a-b884-f8ae5e59dd23 HTTP/1.1", host: "10.0.0.51:25250"In the above example, we can see that the log bundle request results in a tarball size 5408395472 bytes, which is larger than the 5GB max client body size that the director blobstore nginx server is configured with.
In order to change this value to larger than 5GB, perform the following:
1 - SSH onto the BOSH director and become root.
2 - Change to directory /var/vcap/jobs/blobstore/config/sites
3 - make a copy of the blobstore.conf
4 - Open the original blobstore.conf with an editor and increase the value of parameter "client_max_body_size" from 5000m to a larger size (in the above example we see the log bundle is 5.4GB so we would change this value to 6000m which is equal to 6GB) - save the changes.
5 - Restart the blobstore_nginx job with monit restart blobstore_nginx
When the blobstore_nginx job status is running with the new configuration, the log bundle request should succeed.
Feedback
Yes
No
Powered by
