Http 503 EXCEPTION(icap_error)with policy trace showing "Server error: 404 ICAP Service not found"
Article ID: 384973
Updated On:
Products
Issue/Introduction
When attempting to upload a document to a business site users can get some ICAP exception and http 503 status back from the Cloud SWG service.
With a smaller file size the upload works but for some clients the upload fails at a specific threshold.
The page takes a long time (2+ minutes) to show an upload error or display the exception page.
Environment
Cloud SWG with DLP validation of the uploaded document.
Cause
The proxy side error captured with a policy trace shows:
verdict: EXCEPTION(icap_error): Request could not be handled
...
server.response.code: 0
client.response.code: 503
...
ICAP REQMOD Scan Summary:
Error code: server_error
Details: Server error: 404 ICAP Service not found
Summary: icap-error-details: Server error: 404 ICAP Service not found
The fact that there was no ICAP response code indicates that the ICAP request timed-out.
The transaction timings are surprising, or we could also say "inconsitent":
bytes received from client: 13845533
bytes sent to server : 0
bytes received from server: 0
bytes sent to client : 1170
Transaction timing: total-transaction-time 155151 ms
Checkpoint timings:
new-connection: start 1 elapsed 0 ms
client-in: start 185 elapsed 1 ms
client-out: start 155147 elapsed 0 ms
client-out-terminated: start 155147 elapsed 0 ms
access-logging: start 155147 elapsed 4 ms
stop-transaction: start 155151 elapsed 0 ms
Total Policy evaluation time: 5 ms
url_categorization complete time: 0
ICAP Request Scan: create 186, queue-delay n/a, start 186, connect-delay 0, finish 60596
client connection: first-response-byte 0 last-response-byte 155148
access-logging: precompute_fields: 0 ms, logging: 3 ms
The ICAP request is completed in 1 minute indicating a timeout processing the request upload to the ICAP service, yet the total transaction is taking 2 minutes 20 seconds.
A packet capture on the proxy shows that the ICAP upload did not complete within the 60 seconds and looking at the packet capture data transfer we see a very consistent upload speed with some pauses between batches of upload.
Furthermore adding up the captured ip data length we see ~5MBytes were uploaded to the ICAP, which is ~40% of the 13MBytes received from the client. If we check how long it would take to upload the full file at the same rate we have 13/(5/60) = 156 seconds total upload time.
This is the length of the whole transaction so we can safely infer from this data that the transaction duration is driven by the client upload speed to the proxy, and that this upload speed (~735kbps) is insufficient to upload the file up to the service and get an icap response.
Resolution
The packet capture taken on the ICAP port from the proxy to the upstream ICAP services shows that there is no packet loss or bandwidth restriction, and also the constant "poor" upload speed is impacting the total transaction, so it is safe to conclude here that the problem is on the upload path from the client to the proxy.
Feedback
