Error: "com.amazonaws.ResetException: The request to the service failed with a retryable reason" occurs during file uploads.

Products

• VMware Cloud Director

Issue/Introduction

• OSE uploads fail after having completed 100% in the UI.
• An following error is displayed:
  
  The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
  
  
• In /opt/vmware/voss/support/ose-default.log on the OSE server,  you see entries similar to:
  
  2025-05-05 16:00:00 [XNIO-2 task-#] ERROR c.vmware.voss.resource.ResourceBase - Request ########-####-####-####-######## has failed.
  com.amazonaws.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1511)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1284)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
  ...
  ...
  Caused by: java.io.IOException: Resetting to invalid mark
          at java.base/java.io.BufferedInputStream.reset(BufferedInputStream.java:446)

Environment

VMware Object Storage Extension 3.0.0.1

Cause

This issue will occur if there is a network interruption during the upload of the file.

OSE has AWS SDK embedded to transfer data to storage. The AWS SDK has a buffer when streaming data to storage. The max buffer size is the read limit. The default read limit is 1048577B and is not configurable. If there is any network interruption encountered during data transfer, AWS SDK retry is triggered. This relies on the buffer to make retry work. If content size <= max buffer size (the read), retry will work as expected. If content size > max buffer size, it can raise the error that's visible. Mostly it is the limitation of AWS SDK capability.

Resolution

To resolve this issue it's recommended to investigate the network and eliminate any inability issues which would interrupt uploads.

Additional Information

For objects with size <= 100MB, OSE server will leverage regular approach to upload an object.

For objects with size > 100MB, OSE server will leverage transfer manager to upload the object.