Object Storage Extension (OSE) Bucket List and Replication Synchronization Mismatch between Availability Zones

Products

• VMware Cloud Director

Issue/Introduction

VMware Cloud Director Object Storage Extension (OSE) fails to synchronize the bucket list and object data between two Availability Zones (AZ1 and AZ2). While manual sync tasks may report success or hang at specific percentages (e.g., 40%), the global bucket inventory remains inconsistent (e.g., 23 buckets in AZ1 vs. 20 in AZ2). Attempts to create new buckets in one AZ do not reflect in the other, often resulting in "Permission error" or logs indicating the bucket is "not found in local db."

Environment

Cloud Director 10.6.x

OSE 3.x

Cause

The primary cause is a failure in the metadata synchronization layer between OSE instances or an underlying permission mismatch at the S3 storage provider level (ECS). If the OSE local database in AZ2 does not contain the metadata for a bucket created in AZ1, the OSE provider cannot "HEAD" or "GET" the bucket from the storage endpoint, breaking the synchronization logic.

Resolution

 

• Validate S3 Connectivity and Permissions (Out-of-Band):

  • Obtain the S3 credentials for the affected user from the OSE AZ2 portal.

  • Identify the ECS AZ2 endpoint associated with the OSE AZ2 instance.

  • Using an S3 client (e.g., Postman, AWS CLI, or s3cmd), attempt to perform a HEAD and GET request on the missing bucket name directly against the ECS AZ2 endpoint.

  • Result Analysis:

    • If the requests fail with a 403 Forbidden or 404 Not Found at the ECS level, the issue resides in the ECS Cross-Region Replication (CRR) or IAM policy replication.

    • If the requests succeed, the issue is localized to the OSE metadata database.

• Verify OSE Service Logs:

  • Inspect ose-default.log for entries related to "bucket not found in local db" or "403 Access Denied".

  • Ensure the ose-access.log is rotating correctly and capturing current timestamps.

• Lifecycle Management & Upgrade:

  • Note that OSE 3.0 reaches End of General Support (EOGS) in March 2026.

  • Plan an upgrade to OSE 3.1 (Supported until October 2027) to benefit from cumulative hotfixes and improved multi-site sync stability.

 

Additional Information

 

• OSE synchronization is often triggered via the "GO TO BUCKET", "Global Sync" or manual sync button; however, this requires consistent S3 mappings on the backend storage.

• Standard support for OSE 3.0 is expiring; any required hotfixes should be requested via a formal Support Request (SR) while noting that 3.1 is the recommended stable path.