Error: "Upload failed: operation error S3" returned from bosh-blobstore-s3 during Jammy or Windows Stemcell upgrade
Article ID: 434634
Updated On:
Products
Issue/Introduction
When using an S3-compatible blobstore (such as Dell ECS, MinIO, Ceph, Cloudflare R2, Linode Object Storage, certain on-prem gateways, etc.) with the BOSH Director, certain AWS-specific checksum calculation properties introduced in newer bosh-s3cli versions may cause compatibility issues. This typically results in failed blobstore interactions or deployment errors such as XAmzContentSHA256Mismatch. These failures are specifically encountered starting with Jammy stemcells 1.1033 and Windows stemcells 2019.95.
These properties are not exposed through the standard Ops Manager Director configuration API or UI, leading to failed blobstore interactions or deployment errors.
- Specific properties introduced in bosh-s3cli PR #66 requiring manual disablement (for S3-compatible stores that don't support newer AWS SDK checksum algorithms) include:
request_checksum_calculation_enabledresponse_checksum_calculation_enableduploader_request_checksum_calculation_enabledmultipart_upload
- See the this KB for more detailed symptoms encountered during upgrade.
Environment
Elastic Application Runtime (TAS) product, specifically when using an S3 compatible blobstore. This failure is specifically encountered starting with Jammy stemcells 1.1033 and Windows stemcells 2019.95.
Cause
The bosh-s3cli version bump (v0.0.383+) introduced changes to the AWS SDK that enforce stricter checksum handling. S3-compatible storage providers (non-AWS) that do not support these newer AWS SDK checksum algorithms reject the requests. Because Ops Manager does not provide a native toggle for these settings in the GUI, the BOSH Director manifest must be modified using the Director Overrides API.
Resolution
Read this before applying workaround
The suggested solution is to contact your third party s3 vendor and verify when AWS s3 new CRC64NVME checksum algorithm will be supported.
This workaround procedure does not work with BOSH director and only applies to the BOSH Agent. It is recommended to use Opsman 3.2.3 or 3.1.7 which installs a lower stemcell version for the bosh director to workaround this issue. This procedure will enable service tiles other than the director tile to go to the latest stemcell version until a solution can be provided by the third party vendor.
High-Risk Operations & Critical Warnings
Before proceeding, understand how the BOSH Director Overrides API handles different data structures. Improper configuration can lead to a broken BOSH Director and loss of access to your environment.
1. Array vs. Hash Merge Behavior
The Overrides API performs a deep merge for hashes (objects), but entirely replaces arrays.
Manifest Field | Data Type | Merge Behavior | Risk |
blobstore (Director) | Hash | Safe Merge: Only new keys need to be provided; existing keys are preserved. | Low |
agent.env.bosh.blobstores | Array | Full Replace: The entire array is overwritten. You must include all existing connection details. | CRITICAL |
Warning: If you provide only the new checksum properties in the blobstores array without including your existing S3 endpoint, credentials, and bucket name, the BOSH Agent will lose its connection to the blobstore, causing all subsequent deployments and health checks to fail.
2. API Reference
GET /api/v0/staged/director/overrides#---- view current overridesPUT /api/v0/staged/director/overrides#---- replace all overridesDELETE /api/v0/staged/director/overrides#---- remove all overrides
https://<OPSMAN>/docs/#tag/Advanced-Manifest-Configuration
Prerequisites
- For the checksum overrides to take effect, Jammy Stemcell 1.1107 and Windows Stemcell Version 2019.96 or above are required.
- Ops Manager Admin Credentials: Required to generate a UAA token.
- S3 Configuration Details: You must have your current S3 host, bucket name, and access keys ready.
- Terminal Access: curl and python3 (for JSON parsing) are required.
Resolution Steps
Step 1: Obtain a UAA Access Token
Authenticate with the Ops Manager UAA to authorize your API calls.
TOKEN=$(curl -sk https://<OPSMAN>/uaa/oauth/token \ -d "grant_type=password&username=<USERNAME>&password=<PASSWORD>" \ -d "client_id=opsman&client_secret=" \ -H "Content-Type: application/x-www-form-urlencoded" \ | python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])")
Step 2: Enable Advanced Mode
Unlock the infrastructure to allow manifest overrides.
curl -sk https://<OPSMAN>/api/v0/staged/infrastructure/locked \ -X PUT \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"locked": false}'
Step 3: Capture Existing Agent Blobstore Config
Because the blobstores field is an array, you must fetch the current settings to include them in the override payload.
curl -sk https://<OPSMAN>/api/v0/staged/director/manifest \ -H "Authorization: Bearer $TOKEN" \ | python3 -c "import json, sysdata = json.load(sys.stdin)manifest = data.get('manifest', data)for ig in manifest.get('instance_groups', []): if ig.get('name') == 'bosh': blobstores = ig['properties']['agent']['env']['bosh']['blobstores'] print(json.dumps(blobstores, indent=2)) break"
Example output:
json[ { "provider": "s3", "options": { "provider": "s3", "host": "object.ecstestdrive.com", "use_ssl": true, "bucket_name": "my-bucket", "credentials_source": "static", "access_key_id": "my-access-key", "secret_access_key": "((BLOBSTORE_S3_SECRET_ACCESS_KEY))", "port": 443, "signature_version": "2", "host_style": false, "region": "us-east-1" } }]Step 4: Apply Overrides
Construct your PUT request. Merge your existing S3 values (from Step 3) with the new checksum toggles.
curl -sk https://<OPSMAN>/api/v0/staged/director/overrides \
-X PUT \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"overrides": [
{
"section": "instance_groups",
"data": {
"agent": {
"env": {
"bosh": {
"blobstores": [
{
"provider": "s3",
"options": {
"provider": "s3",
"host": "<YOUR_HOST>",
"bucket_name": "<YOUR_BUCKET>",
"access_key_id": "<YOUR_KEY>",
"secret_access_key": "((BLOBSTORE_S3_SECRET_ACCESS_KEY))",
"request_checksum_calculation_enabled": false,
"response_checksum_calculation_enabled": false,
"uploader_request_checksum_calculation_enabled": false,
"multipart_upload": false,
"use_ssl": true,
"port": 443,
"signature_version": "2",
"region": "us-east-1"
}
}
]
}
}
}
}
}
]
}'
Step 5: Verify and Re-lock
Verify the manifest contains the new values:
curl -sk https://<OPSMAN>/api/v0/staged/director/manifest \ -H "Authorization: Bearer $TOKEN" \ | python3 -c "import json, sysdata = json.load(sys.stdin)manifest = data.get('manifest', data)for ig in manifest.get('instance_groups', []): if ig.get('name') == 'bosh': props = ig['properties'] print(json.dumps(props['agent']['env']['bosh']['blobstores'], indent=2)) break"
Re-lock the infrastructure:
curl -sk https://<OPSMAN>/api/v0/staged/infrastructure/locked \ -X PUT \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"locked": true}'
Step 6: Apply Changes
Finalize the configuration by clicking Apply Changes in the Ops Manager UI.
Removing Overrides
If you need to revert to the default Ops Manager-generated configuration, use the DELETE method:
curl -sk https://<OPSMAN>/api/v0/staged/director/overrides \ -X DELETE \ -H "Authorization: Bearer $TOKEN"
Feedback
