bosh-blobstore-s3 error on jammy stemcells 1.1033 and Windows stemcells 2019.95 and later
search cancel

bosh-blobstore-s3 error on jammy stemcells 1.1033 and Windows stemcells 2019.95 and later

book

Article ID: 430821

calendar_today

Updated On:

Products

VMware Tanzu Application Service

Issue/Introduction

When we tried to do upgrade to 10.2.8, then got error below. We checked communication with ECS and no issue.

Apply Changes log output:

Task 224665 | 23:29:14 | Preparing deployment: Preparing deployment (00:00:08)
Task 224665 | 23:29:22 | Preparing deployment: Rendering templates (00:02:15)
Task 224665 | 23:31:38 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 224665 | 23:31:38 | Compiling packages: golang-1-linux/########################################
Task 224665 | 23:31:38 | Compiling packages: golang-1.25-linux/########################################
Task 224665 | 23:31:38 | Compiling packages: defender/######################################## (00:01:17)
                       L Error: Action Failed get_task: Task ######################################## result: Compiling package defender: Uploading compiled package: Creating blob in inner blobstore: Making put command: Shelling out to bosh-blobstore-s3 cli: Running command: 'bosh-blobstore-s3 -c /var/vcap/bosh/etc/blobstore-s3.json put /var/vcap/data/tmp/bosh-platform-disk-TarballCompressor-CompressSpecificFilesInDir########################################', stdout: '', stderr: '2026/02/24 23:32:43 Upload failed: operation error S3: PutObject, https response error StatusCode: 400, RequestID: ########################################, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
2026/02/24 23:32:43 performing operation put: upload failure: operation error S3: PutObject, https response error StatusCode: 400, RequestID: ########################################, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
': exit status 1
Task 224665 | 23:33:35 | Compiling packages: golang-1.25-linux/######################################## (00:01:57)
                       L Error: Action Failed get_task: Task ######################################## result: Compiling package golang-1.25-linux: Uploading compiled package: Creating blob in inner blobstore: Making put command: Shelling out to bosh-blobstore-s3 cli: Running command: 'bosh-blobstore-s3 -c /var/vcap/bosh/etc/blobstore-s3.json put /var/vcap/data/tmp/bosh-platform-disk-TarballCompressor-CompressSpecificFilesInDir4033261733 ########################################', stdout: '', stderr: '2026/02/24 23:33:24 Upload retry limit exceeded: upload multipart failed, upload id: ########################################, cause: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: ########################################, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
2026/02/24 23:33:24 performing operation put: upload retry limit exceeded: upload multipart failed, upload id: ########################################, cause: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: ########################################, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
Updating deployment:
  Expected task '224665' to succeed but state is 'error'
Exit code 1
': exit status 1
Task 224665 | 23:33:36 | Compiling packages: golang-1-linux/######################################## (00:01:58)
                       L Error: Action Failed get_task: Task ######################################## result: Compiling package golang-1-linux: Uploading compiled package: Creating blob in inner blobstore: Making put command: Shelling out to bosh-blobstore-s3 cli: Running command: 'bosh-blobstore-s3 -c /var/vcap/bosh/etc/blobstore-s3.json put /var/vcap/data/tmp/bosh-platform-disk-TarballCompressor-CompressSpecificFilesInDir3200310843 ########################################', stdout: '', stderr: '2026/02/24 23:33:24 Upload retry limit exceeded: upload multipart failed, upload id: ########################################, cause: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: ########################################, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
2026/02/24 23:33:24 performing operation put: upload retry limit exceeded: upload multipart failed, upload id: ########################################, cause: operation error S3: UploadPart, https response error StatusCode: 400, RequestID: ########################################, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
': exit status 1
Task 224665 | 23:33:36 | Error: Action Failed get_task: Task ######################################## result: Compiling package defender: Uploading compiled package: Creating blob in inner blobstore: Making put command: Shelling out to bosh-blobstore-s3 cli: Running command: 'bosh-blobstore-s3 -c /var/vcap/bosh/etc/blobstore-s3.json put /var/vcap/data/tmp/bosh-platform-disk-TarballCompressor-CompressSpecificFilesInDir2191962756 ########################################', stdout: '', stderr: '2026/02/24 23:32:43 Upload failed: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 0a8b31d3:19c78681192:98b8:e14b, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
2026/02/24 23:32:43 performing operation put: upload failure: operation error S3: PutObject, https response error StatusCode: 400, RequestID: ########################################, HostID: ########################################, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
': exit status 1

Environment

Jammy stemcells 1.1033 and later

Windows stemcells 2019.95 and later

Cause

Recent stemcell update ubuntu-jammy v1.1033 bumped the bosh-s3cli version to v0.0.383 (or later). The same bosh-s3cli bump is applied in Windows stemcell 2019.95 (or later). This version includes changes to the AWS SDK or signing logic that enforce stricter checksum handling for SigV4 requests. Many S3-compatible storage providers (MinIO, Ceph, Cloudflare R2, Linode Object Storage, certain on-prem gateways, etc.) do not fully support or correctly process these newer integrity checks/headers, leading to the mismatch rejection—even though the same setup worked before the bump.AWS-native S3 usually handles it fine, but non-AWS or older/partial implementations often fail.

Resolution

Workarounds:

  • Use an earlier version of the ubuntu-jammy stemcell (e.g., 1.1018), or Windows stemcell (e.g., 2019.94)
  • Downgrade or Pin the bosh-s3cli Version

This issue is tracked in the Cloud Foundry project: github.com/cloudfoundry/bosh-s3cli/issues/65.
Users report success by forcing an older bosh-s3cli version (pre-0.0.383, e.g., v0.0.382 or earlier) in the stemcell or BOSH director configuration until a patched release lands.

Engineering has provided a resolution that explains and addresses the root cause.

Powered by Wolken Software