Error "Invalid characters/missing colon in header line..." message appears when accessing some websites after upgrading the ProxySG to SGOS 6.5.9.2 or later

Products

• Asset Management Solution
• Data Center Security Monitoring Edition
• ProxySG Software - SGOS

Issue/Introduction

When accessing some websites, users receive the following error exception message:

Server response could not be processed. Invalid characters/missing colon in header line or header name is empty
This could be caused by a malformed response, or possibly a misconfiguration.

Cause

Users receive this error message because the site they are attempting to access is not RFC compliant. To allow users access to legitimate sites that are not RFC complaint, Blue Coat has added support for detecting malformed HTTP response headers in SGOS 6.5.9.2 and later.
These changes include:

• Converting alternate whitespace characters in headers to standard spaces
• Improved handling of invalid characters at the beginning of header and HTTP 0.9 responses
• Detecting invalid HTTP version strings in HTTP responses.
• Improved handling of invalid/missing HTTP response codes.
• Unfolding normal and empty continuation lines in the HTTP response

Improved handling of different variations of responses that include chunked encoding instructions.
 

Resolution

If you see the above error exception message, Blue Coat recommends that you upgrade to SGOS 6.5.9.11. SGOS 6.5.9.11 has support for normalizing certain forms of invalid headers rather than rejecting them. Two new access log fields with policy substitution (x-bluecoat-normalized-response-headers and x-bluecoat-invalid-response-headers) have also been added. In cases where 6.5.9.11 does still reject a response, the x-bluecoat-invalid-response-headers field will report what made the response invalid. And in cases where the ProxySG appliance automatically normalizes the headers and returns the corrected response to the client, the x-bluecoat-normalized-response-headers field reports what normalization changes the appliance made.

If you continue to see the exception error after upgrading to 6.5.9.11, you can enable the following CPL to bypass the exception:
Note: If you enable the ProxySG appliance to tolerate invalid headers, your appliances might be open to client-side attacks that involve headers that are not RFC compliant. Blue Coat recommends that you should only enable the following policy in cases where you trust the OCS and the network path between the OCS and you client computers.

<proxy>      
   response.raw_headers.tolerate(invalid_header)

To trigger by domain names you want to tolerate invalid header  response, you could use the following CPL:
 

<proxy>
     condition=tolerate_sites_resp response.raw_headers.tolerate(invalid_header)

define url.domain condition tolerate_sites_resp
    domain_1.com
    domain_2.com
    www.domain_3.com
end