Customer experience issues with connectivity when sending data to AWS S3 server via Proxy as spotted in the Syslog.
SYSLOG Collector:
Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 141688 <CUSTOMER-SERVER-IP> 0 TUNNELED 47602 124854591 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY IP> unavailable 7a85de92a6af4ccc-00000000dc311c8a-000000006369f62b - - -
Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 113195 <CUSTOMER-SERVER-IP> 0 TUNNELED 34930 96270471 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY-IP> unavailable 7a85de92a6af4ccc-00000000dc311c8b-000000006369f648 -
9 13:00:05,432 [user:] [pipeline:] [runner:] [thread:webserver-2241303] [stage:] INFO Authenticator - Authentication succeeded for dn: CN=fc8diadmin,OU=ServiceAccounts,OU=ADM,OU=LFC,DC=gfoundries,DC=com
2022-11-09 13:00:06,311 [user:***] [pipeline:FDC_TRACE_CLOUD_PROD/FDCTRACE23352060-455a-49e6-8492-a197e366cc85] [runner:] [thread:ProductionPipelineRunnable-FDCTRACE23352060-455a-49e6-8492-a197e366cc85-FDC_TRACE_CLOUD_PROD] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.pipeline.api.StageException: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: DC8RN0FVQF8YDEMZ; S3 Extended Request ID: 2JfYCeJ34tYtdTd3RikE97IMD2H/s/KTefg/nYTKxmrZohea6IG2s9qn2S5hB9suzzVgmPrwIeM=; Proxy:
Release : 6.7.5.17
- Proxy is not blocking anything for the application data transfer;
- Issue with Amazon AWS S3 server application
SYSLOG1 MESSAGE:
9 13:00:05,432 [user:] [pipeline:] [runner:] [thread:webserver-2241303] [stage:] INFO Authenticator - Authentication succeeded for dn: CN=<name>,OU=ServiceAccounts,OU=ADM,OU=LFC,DC=<name>,DC=com
2022-11-09 13:00:06,311 [user:***] [pipeline:FDC_TRACE_CLOUD_PROD/FDCTRACE23352060-455a-49e6-8492-a197e366cc85] [runner:] [thread:ProductionPipelineRunnable-FDCTRACE23352060-455a-49e6-8492-a197e366cc85-FDC_TRACE_CLOUD_PROD] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.pipeline.api.StageException: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: DC8RN0FVQF8YDEMZ; S3 Extended Request ID: 2JfYCeJ34tYtdTd3RikE97IMD2H/s/KTefg/nYTKxmrZohea6IG2s9qn2S5hB9suzzVgmPrwIeM=; Proxy:
The error code 500 Internal Error indicates that Amazon S3 can't handle the request at that time. The error code 503 Slow Down typically indicates that the number of requests to your S3 bucket is very high, exceeding the request rate.
This is not network Proxy related issue, the number of connection to S3 Bucket is exceeded and overloaded that’s why it produces the error and breaks the communication - AmazonS3Exception: Internal Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Error; Request ID: A4DBBEXAMPLE2C4D)
SYSLOG2 MESSAGE:
Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 141688 <CUSTOMER-SERVER-IP> 0 TUNNELED 47602 124854591 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY IP> unavailable 7a85de92a6af4ccc-00000000dc311c8a-000000006369f62b - - -
Nov 8 06:27:05 <PUBLIC-IP> 2022-11-08 06:27:05 113195 <CUSTOMER-SERVER-IP> 0 TUNNELED 34930 96270471 unknown ssl <Amazon Server> 443 / - - - <Amazon Server> <AMAZON SERVER-IP> "None" - - - - OBSERVED "Cloud Infrastructure" - <PROXY-IP> unavailable 7a85de92a6af4ccc-00000000dc311c8b-000000006369f648 -
Client IP tunneled application data to <Amazon Server> via Proxy and got the response back from <AWS-SERVER-IP> with unavailable code - (HTTP) 503 Service Unavailable server error response
AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down; Request ID: A4DBBEXAMPLE2C4D)
AWS issue with bucket performance