Spring Cloud Dataflow Stream using s3 storage failed to connect to service endpoint

Products

• VMware Tanzu Spring Runtime

Issue/Introduction

The stream failed with the error:

2024-08-27 15:13:37.019  WARN [s3-source,,] 64263 --- [           main] i.InstanceMetadataServiceResourceFetcher : Fail to retrieve token com.amazonaws.SdkClientException: Failed to connect to service endpoint:

Environment

Spring Cloud Dataflow 

Stream Source or Sink using AWS S3 

 

Cause

These are the possible causes:

1. The endpoint-url is invalid.

2. Credential are invalid.

Resolution

Resolved with the following:

1. Check and verify endpoint_url

Identify which Amazon S3 endpoints to use based on the Region and Protocol

 

2. Verify credentials by connecting to the S3 console using the cAWS S3 credentials 

 

Example:

1. Create stream using. Copy and paste this to the Spring Cloud Dataflow dashboard - stream create

s3 --endpoint-url=https://s3.us-east-1.amazonaws.com --path-style-access=true --local-dir=/path/mydir --remote-dir=mybucket --mode=lines | log

2. Deploy stream application with the following using Freetext to add the credentials (app.s3.cloud.aws.*)

app.s3.endpoint-url=******

app.s3.file.consumer.mode=lines

app.s3.local-dir=/Users/bsamson/download

app.s3.path-style-access=true

app.s3.remote-dir=mybucket

app.s3.cloud.aws.credentials.accessKey=MY_ACCESS_KEY

app.s3.cloud.aws.credentials.secretKey=MY_SECRET_KEY

app.s3.cloud.aws.region.static=us-east-1

app.s3.cloud.aws.stack.auto=false

 

Note: You can test streams using s3 using minio.

Additional Information

Reference documents:

https://github.com/spring-cloud/stream-applications/blob/main/applications/source/s3-source/README.adoc 

https://github.com/spring-cloud/stream-applications/blob/main/applications/sink/s3-sink/README.adoc