Error: "Upload failed: operation error S3" returned from bosh-blobstore-s3 during Jammy or Windows Stemcell upgrade
search cancel

Error: "Upload failed: operation error S3" returned from bosh-blobstore-s3 during Jammy or Windows Stemcell upgrade

book

Article ID: 434634

calendar_today

Updated On:

Products

VMware Tanzu Application Service

Issue/Introduction

When using an S3-compatible blobstore (such as Dell ECS, MinIO, Ceph, Cloudflare R2, Linode Object Storage, certain on-prem gateways, etc.) with the BOSH Director, certain AWS-specific checksum calculation properties introduced in newer bosh-s3cli versions may cause compatibility issues. This typically results in failed blobstore interactions or deployment errors such as XAmzContentSHA256Mismatch. These failures are specifically encountered starting with Jammy stemcells 1.1033 and Windows stemcells 2019.95.

These properties are not exposed through the standard Ops Manager Director configuration API or UI, leading to failed blobstore interactions or deployment errors.

  • Specific properties introduced in bosh-s3cli PR #66 requiring manual disablement (for S3-compatible stores that don't support newer AWS SDK checksum algorithms) include:
    • request_checksum_calculation_enabled
    • response_checksum_calculation_enabled
    • uploader_request_checksum_calculation_enabled
    • multipart_upload
  • See the this KB for more detailed symptoms encountered during upgrade.

Environment

Elastic Application Runtime (TAS) product, specifically when using an S3 compatible blobstore. This failure is specifically encountered starting with Jammy stemcells 1.1033 and Windows stemcells 2019.95.

Cause

The bosh-s3cli version bump (v0.0.383+) introduced changes to the AWS SDK that enforce stricter checksum handling. S3-compatible storage providers (non-AWS) that do not support these newer AWS SDK checksum algorithms reject the requests. Because Ops Manager does not provide a native toggle for these settings in the GUI, the BOSH Director manifest must be modified using the Director Overrides API.

Resolution

Read this before applying workaround

The suggested solution is to contact your third party s3 vendor and verify when AWS s3 new CRC64NVME checksum algorithm will be supported.  

This workaround procedure does not work with BOSH director and only applies to the BOSH Agent.  It is recommended to use Opsman 3.2.3 or 3.1.7 which installs a lower stemcell version for the bosh director to workaround this issue.  This procedure will enable service tiles other than the director tile to go to the latest stemcell version until a solution can be provided by the third party vendor. 

 

High-Risk Operations & Critical Warnings

Before proceeding, understand how the BOSH Director Overrides API handles different data structures. Improper configuration can lead to a broken BOSH Director and loss of access to your environment.

 

1. Array vs. Hash Merge Behavior

The Overrides API performs a deep merge for hashes (objects), but entirely replaces arrays.

Manifest Field

Data Type

Merge Behavior

Risk

blobstore (Director)

Hash

Safe Merge: Only new keys need to be provided; existing keys are preserved.

Low

agent.env.bosh.blobstores

Array

Full Replace: The entire array is overwritten. You must include all existing connection details.

CRITICAL

 

 

 

 

 

 









Warning: If you provide only the new checksum properties in the blobstores array without including your existing S3 endpoint, credentials, and bucket name, the BOSH Agent will lose its connection to the blobstore, causing all subsequent deployments and health checks to fail.

 

2. API Reference

    • GET /api/v0/staged/director/overrides   #---- view current overrides
    • PUT /api/v0/staged/director/overrides  #---- replace all overrides
    • DELETE /api/v0/staged/director/overrides  #---- remove all overrides
 
Full documentation: https://<OPSMAN>/docs/#tag/Advanced-Manifest-Configuration

 

Prerequisites

  • For the checksum overrides to take effect, Jammy Stemcell 1.1107 and Windows Stemcell Version 2019.96 or above are required.
  • Ops Manager Admin Credentials: Required to generate a UAA token.
  • S3 Configuration Details: You must have your current S3 host, bucket name, and access keys ready.
  • Terminal Access: curl and python3 (for JSON parsing) are required.

 

 

Resolution Steps

 

Step 1: Obtain a UAA Access Token

Authenticate with the Ops Manager UAA to authorize your API calls.

TOKEN=$(curl -sk https://<OPSMAN>/uaa/oauth/token \
  -d "grant_type=password&username=<USERNAME>&password=<PASSWORD>" \
  -d "client_id=opsman&client_secret=" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])")

 

Step 2: Enable Advanced Mode

Unlock the infrastructure to allow manifest overrides.

curl -sk https://<OPSMAN>/api/v0/staged/infrastructure/locked \
  -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"locked": false}'

 

Step 3: Capture Existing Agent Blobstore Config

Because the blobstores field is an array, you must fetch the current settings to include them in the override payload.

curl -sk https://<OPSMAN>/api/v0/staged/director/manifest \
  -H "Authorization: Bearer $TOKEN" \
  | python3 -c "
import json, sys
data = json.load(sys.stdin)
manifest = data.get('manifest', data)
for ig in manifest.get('instance_groups', []):
    if ig.get('name') == 'bosh':
        blobstores = ig['properties']['agent']['env']['bosh']['blobstores']
        print(json.dumps(blobstores, indent=2))
        break
"

 

Example output:

json
[
  {
    "provider": "s3",
    "options": {
      "provider": "s3",
      "host": "object.ecstestdrive.com",
      "use_ssl": true,
      "bucket_name": "my-bucket",
      "credentials_source": "static",
      "access_key_id": "my-access-key",
      "secret_access_key": "((BLOBSTORE_S3_SECRET_ACCESS_KEY))",
      "port": 443,
      "signature_version": "2",
      "host_style": false,
      "region": "us-east-1"
    }
  }
]
 
 

Step 4: Apply Overrides

Construct your PUT request. Merge your existing S3 values (from Step 3) with the new checksum toggles.

curl -sk https://<OPSMAN>/api/v0/staged/director/overrides \
  -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "overrides": [
      {
        "section": "instance_groups",
        "data": {
          "agent": {
            "env": {
              "bosh": {
                "blobstores": [
                  {
                    "provider": "s3",
                    "options": {
                      "provider": "s3",
                      "host": "<YOUR_HOST>",
                      "bucket_name": "<YOUR_BUCKET>",
                      "access_key_id": "<YOUR_KEY>",
                      "secret_access_key": "((BLOBSTORE_S3_SECRET_ACCESS_KEY))",
                      "request_checksum_calculation_enabled": false,
                      "response_checksum_calculation_enabled": false,
                      "uploader_request_checksum_calculation_enabled": false,
                      "multipart_upload": false,
                      "use_ssl": true,
                      "port": 443,
                      "signature_version": "2",
                      "region": "us-east-1"
                    }
                  }
                ]
              }
            }
          }
        }
      }
    ]
  }'

 

Step 5: Verify and Re-lock

Verify the manifest contains the new values:

curl -sk https://<OPSMAN>/api/v0/staged/director/manifest \
  -H "Authorization: Bearer $TOKEN" \
  | python3 -c "
import json, sys
data = json.load(sys.stdin)
manifest = data.get('manifest', data)
for ig in manifest.get('instance_groups', []):
    if ig.get('name') == 'bosh':
        props = ig['properties']
        print(json.dumps(props['agent']['env']['bosh']['blobstores'], indent=2))
        break
"

 

Re-lock the infrastructure:

curl -sk https://<OPSMAN>/api/v0/staged/infrastructure/locked \
  -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"locked": true}'

 

 

Step 6: Apply Changes

Finalize the configuration by clicking Apply Changes in the Ops Manager UI. 

 
WARNING: If you only provide the new checksum properties in the agent blobstores array without the existing S3 connection settings, the deploy will fail because the agent won't have endpoint, credentials, or bucket information.
Review the diff carefully: In the REVIEW PENDING CHANGES > SEE CHANGES of the tile in Opsman GUI, if you see existing S3 properties being removed (prefixed with -), cancel the deploy and fix your override payload.

 

Removing Overrides

If you need to revert to the default Ops Manager-generated configuration, use the DELETE method:

curl -sk https://<OPSMAN>/api/v0/staged/director/overrides \
  -X DELETE \
  -H "Authorization: Bearer $TOKEN"

Powered by Wolken Software