BOSH backup fails with TotalPartsExceeded error

Products

• Pivotal Concourse

Issue/Introduction

Symptoms:
When using either BOSH Backup and Restore (BBR) or Concourse bbr-pcf-pipeline-tasks to backup the BOSH director or deployments to MINIO S3 blobstore system, it might fail with following error messages:

error running command: MultipartUpload: upload multipart failed
upload id: 6142565f-###-###-####-f20dfc36edae
caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit

Environment

Cause

Both BBR and Concourse bbr-pcf-pipeline-tasks are utilizing the MultipartUpload functionality of AWS SDK GO library to upload a very large file to S3 blobstore. However they each handle it in different way.

BBR will do MulitipartUpload when the file size is greater than 100MB (changed to 1GB since backup-and-restore-sdk-release v1.11.2). However, it enforces part-size to be 100MB with the maximum allowed number of parts being 10000. This means part-size is not configurable. So the hard limit for maximum file size to be uploaded to S3 blobstore, with MultipartUpload, is 1000GB.

Concourse bbr-pcf-pipeline-tasks (since v3.9) will automatically adjust part-size according to file size. This should prevent the TotalPartsExceeded error. 

However, the TotalPartsExceeded error has been seen with both BBR and Concourse bbr-pcf-pipeline-task, the file size being less than 100GB in all these cases. This symptom could be explained by the fact that some S3 blobstore systems, like MINIO, have known issues that their API implementation is either not respecting part-size or it could be treating folders inside the bucket as a single file. Hence a fake TotalPartsExceeded is reported.

Resolution

If it's confirmed that the issue is not with either BBR (the file size of less than 1000GB) or Concourse bbr-pcf-pipeline-tasks, then updating the MINIO blobstore system to the latest version is recommended to get necessary bug fixes about correctly treating part-size or folders. As of the time this article is published, the latest version of MINIO is 2018-12-27T18:33:08Z.