"error: RPC failed; HTTP 503 curl 22 The requested URL returned error: 503" error downloading files from github

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Users accessing internet services via Cloud SWG using proxy forwarding access method.

Cloud SWG policies managed using Management Center.

Some users report getting 503 errors using GIT CLI command as shown below:

user@host:/mnt/c/Dev$ GIT_SSL_NO_VERIFY=1 git clone -v https://github.com/example/test-repo test
Cloning into 'test'...
POST git-upload-pack (gzip 83115 to 41507 bytes)
error: RPC failed; HTTP 503 curl 22 The requested URL returned error: 503
fatal: the remote end hung up unexpectedly

Access logs for the above query show the 503 status with a 'tcp_error' verdict

A large number of developers use git to sync up project files but only a handful report this issue.

Environment

Proxy forwarding.

Downloading files from github.

Management Center/UPE.

Cause

Large file downloads taking more than 3 minutes to complete and hitting HTTP timeout.

Resolution

Increase the http receive timeout for the github.com domain using the following CPL code:

<proxy>
url.domain="github.com" http.server.recv.timeout(360)

Additional Information

The git CLI command can be run in debug mode to gather more verbose output (and confirm TCP errors from the Cloud proxy side) using the following:

GIT_CURL_VERBOSE=1 GIT_SSL_NO_VERIFY=true git clone https://github.com/example/test-repo)

Using the proxy access logs, we could see the 'time taken' field for all tcp_error 503 errors to github.com were 180 seconds.

Although the files were initially claimed to be small, we could see that most were between 1.5GB and 3GB is size.

A sample policy trace from the proxy when the issue happened confirmed files sizes and timeouts too:

bytes received from client: 43981
bytes sent to server      : 43932
bytes received from server: 1976394943
bytes sent to client      : 0
 
This causes a timeout seen after 180 seconds
 
Transaction timing: total-transaction-time 180170 ms
Checkpoint timings:
new-connection: start 75 elapsed 0 ms
client-in: start 76 elapsed 1 ms
server-out: start 77 elapsed 0 ms
server-in-ssl: start 78 elapsed 1 ms
server-in: start 10189 elapsed 0 ms
client-out-terminated: start 180130 elapsed 1 ms
access-logging: start 180132 elapsed 0 ms
stop-transaction: start 180170 elapsed 0 ms