Uploading multiple files concurrently using OSE hitting a concurrency limit

Products

• VMware Cloud Director

Issue/Introduction

• Issue occurring when uploading multiple files using a script. The ose-default.log then has the following error:
  
  No sufficient queue for async tasks

• Users of VMware Cloud Director Object Storage Extension (OSE) 3.1.x may experience intermittent failures during high-concurrency object operations, such as large-scale uploads or deletions via automation scripts (e.g., Loki). 

• The ose-default.log file report the following error: 
  
  WARNING - Attempt 1 failed for file_921.txt: Failed to upload ./txt_files/file_921.txt to #######-435lkt/file_921.txt: An error occurred (ERROR_NO_SUFFICIENT_ASYNC_TASK_QUEUE) when calling the PutObject operation (reached max retries: 4): No sufficient queue for async tasks.
2025-07-17 14:20:05 [XNIO-2 task-83] ERROR c.v.voss.service.async.AsyncTasks - Submissions has exceeded 16 times of ForkJoin pool size: 513. 

• We can also see NoSuchKey' messages:

  2025-07-17 14:18:01 [XNIO-2 task-154] ERROR c.vmware.voss.resource.ResourceBase - Request ########-####-####-####-############ has failed.com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ########-####-####-####-############; S3 Extended Request ID: ################################; Proxy: null)        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418

   

Environment

• VMware Cloud Director 10.6.1
• VMware Cloud Director Object Storage Extension (OSE) 3.1.x

Cause

The issue is caused by the saturation of the internal Java ForkJoinPool used by OSE to manage asynchronous metadata synchronization between the storage platform (e.g., Cloudian) and the OSE database.

Resolution

This issue is under review by SE (Sustaining Engineering).

Subscribto this knowledge article to get updates on this issue.