"Error with blobstore: Excon::Error::Socket - HTTP/1.1 411 Length Required" is returned when pushing an application

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When customer is using Signature Version 4 with some S3 compatible blobstore providers, like Hitachi for TAS blobstore. cf push will fail with "Error with blobstore: Excon::Error::Socket - HTTP/1.1 411 Length Required" 

Similar message "You must provide the Content-Length HTTP header." could also be seen in cloud_controller_ng.log file. For example,

{"timestamp":"2023-04-12T02:02:36.476547553Z","message":"Error with blobstore: Excon::Error::Socket - HTTP/1.1 411 Length Required\r\nDate: Wed, 12 Apr 2023 02:03:21 GMT\r\nContent-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';\r\nCache-Control: no-cache,no-store,must-revalidate\r\nX-Download-Options: noopen\r\nStrict-Transport-Security: max-age=31536000; includeSubDomains\r\nX-Frame-Options: SAMEORIGIN\r\nPragma: no-cache\r\nVary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method\r\nX-XSS-Protection: 1; mode=block\r\nExpires: Thu, 01 Jan 1970 00:00:00 GMT\r\nX-DNS-Prefetch-Control: off\r\nX-Content-Type-Options: nosniff\r\nContent-Type: application/xml;charset=utf-8\r\nTransfer-Encoding: chunked\r\n\r\n102\r\n<?xml version='1.0' encoding='UTF-8'?>\n<Error>\n <Code>MissingContentLength</Code>\n <Message>You must provide the Content-Length HTTP header.</Message>\n <RequestId>1681265001087</RequestId>\n <HostId>XXX</HostId>\n</Error>\n\n\r\n0\r\n\r\nBroken pipe (Excon::Error)","log_level":"error","source":"cc.error_handling_client","data":{"request_guid":"XXX"},"thread_id":89340,"fiber_id":89400,"process_id":7,"file":"/var/vcap/data/packages/cloud_controller_ng/17ce217254f92741c85c7119af2dc3cfd5652ed7/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb","lineno":67,"method":"rescue in error_handling"}

Resolution

This is because certain S3 compatible blobstore providers, like Oracle and Hitachi, do not support signature v4 streaming. Starting from TAS version 2.11.38/2.13.20/3.0.10/4.0.1, it allows operators to disable signature v4 streaming which will enable customer to use blobstores like Oracle and Hitachi.