Amazon AWS S3 returns 500 & 503 HTTP Error when sending data via Proxy
search cancel

Amazon AWS S3 returns 500 & 503 HTTP Error when sending data via Proxy

book

Article ID: 254022

calendar_today

Updated On:

Products

SG-S500 ProxySG Software - SGOS ISG Proxy

Issue/Introduction

Customer experience issues with connectivity when sending data to AWS S3 server via Proxy as spotted in the Syslog.

  • Policy is set to Allow all URLs for category Cloud Infrastructure for the AWS domains, servers are allowed to communicate;
  • The communication seen in Wireshark goes through without an issue, only from time to time got the issue from AWS side;
  • Checked with the Policy there is nothing blocked by Proxy for the AWS services

 

SYSLOG Collector:

Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 141688 <CUSTOMER-SERVER-IP> 0 TUNNELED 47602 124854591 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY IP> unavailable 7a85de92a6af4ccc-00000000dc311c8a-000000006369f62b - - -

Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 113195 <CUSTOMER-SERVER-IP> 0 TUNNELED 34930 96270471 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY-IP> unavailable 7a85de92a6af4ccc-00000000dc311c8b-000000006369f648 -  

9 13:00:05,432 [user:] [pipeline:] [runner:] [thread:webserver-2241303] [stage:] INFO  Authenticator - Authentication succeeded for dn: CN=fc8diadmin,OU=ServiceAccounts,OU=ADM,OU=LFC,DC=gfoundries,DC=com

2022-11-09 13:00:06,311 [user:***] [pipeline:FDC_TRACE_CLOUD_PROD/FDCTRACE23352060-455a-49e6-8492-a197e366cc85] [runner:] [thread:ProductionPipelineRunnable-FDCTRACE23352060-455a-49e6-8492-a197e366cc85-FDC_TRACE_CLOUD_PROD] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.pipeline.api.StageException: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: DC8RN0FVQF8YDEMZ; S3 Extended Request ID: 2JfYCeJ34tYtdTd3RikE97IMD2H/s/KTefg/nYTKxmrZohea6IG2s9qn2S5hB9suzzVgmPrwIeM=; Proxy:

Environment

Release : 6.7.5.17

Cause

- Proxy is not blocking anything for the application data transfer;

- Issue with Amazon AWS S3 server application

 

SYSLOG1 MESSAGE:

9 13:00:05,432 [user:] [pipeline:] [runner:] [thread:webserver-2241303] [stage:] INFO  Authenticator - Authentication succeeded for dn: CN=<name>,OU=ServiceAccounts,OU=ADM,OU=LFC,DC=<name>,DC=com

2022-11-09 13:00:06,311 [user:***] [pipeline:FDC_TRACE_CLOUD_PROD/FDCTRACE23352060-455a-49e6-8492-a197e366cc85] [runner:] [thread:ProductionPipelineRunnable-FDCTRACE23352060-455a-49e6-8492-a197e366cc85-FDC_TRACE_CLOUD_PROD] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.pipeline.api.StageException: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: DC8RN0FVQF8YDEMZ; S3 Extended Request ID: 2JfYCeJ34tYtdTd3RikE97IMD2H/s/KTefg/nYTKxmrZohea6IG2s9qn2S5hB9suzzVgmPrwIeM=; Proxy:

The error code 500 Internal Error indicates that Amazon S3 can't handle the request at that time. The error code 503 Slow Down typically indicates that the number of requests to your S3 bucket is very high, exceeding the request rate.

This is not network Proxy related issue, the number of connection to S3 Bucket is exceeded and overloaded that’s why it produces the error and breaks the communication - AmazonS3Exception: Internal Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Error; Request ID: A4DBBEXAMPLE2C4D)

SYSLOG2 MESSAGE:

Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 141688 <CUSTOMER-SERVER-IP> 0 TUNNELED 47602 124854591 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY IP> unavailable 7a85de92a6af4ccc-00000000dc311c8a-000000006369f62b - - -

Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 113195 <CUSTOMER-SERVER-IP> 0 TUNNELED 34930 96270471 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY-IP> unavailable 7a85de92a6af4ccc-00000000dc311c8b-000000006369f648 -  

Client IP  tunneled application data to <Amazon Server> via Proxy and got the response back from <AWS-SERVER-IP> with unavailable code -  (HTTP) 503 Service Unavailable server error response

AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down; Request ID: A4DBBEXAMPLE2C4D)

Resolution

AWS issue with bucket performance

  • AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down; Request ID: A4DBBEXAMPLE2C4D)
  • AmazonS3Exception: Internal Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Error; Request ID: A4DBBEXAMPLE2C4D)

 

Powered by Wolken Software