search cancel

Splunk server is seeing unknown characters when trying to process proxy access log.

book

Article ID: 216780

calendar_today

Updated On:

Products

ProxySG Software - SGOS

Issue/Introduction

There is an issue with the Splunk server when trying to process the proxy access logs.

The access log been parsed to the Splunk server contains unknown characters.

Environment

Release : 6.7

Component : Proxy access logs.

Cause

The issue is caused by an intermediate device or the Splunk server itself interprets the TCP headers as part of the data.

 

Below is from the Splunk server and the bolded parts is the issue:

<111>1 2021-05-26T06:33:08 xxxxx bluecoat - splunk_format - c-ip=xxxxx rs-Content-Type="text/html" cs-auth-groups=- cs-bytes=926 cs-categories="Finance" cs-host=xxxxx cs-ip=xxxxx cs-method=POST cs-uri-port=443 cs-uri-scheme=https cs-User-Agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" cs-username=xxxxx dnslookup-time=0 duration=0 rs-status=200 rs-version=HTTP/1.1 s-action=TCP_NC_MISS s-ip=xxxxx service.name=xxxxx service.group="Standard" s-supplier-ip=xxxxx s-supplier-name=xxxxx sc-bytes=15759 sc-filter-result=OBSERVED sc-status=200 time-taken=297 x-exception-id=- x-virus-id=- c-url=xxxxx cs-Referer=xxxxx c-cpu=- connect-time=76 cs-auth-groups=- cs-headerlength=918 cs-threat-risk=2 r-ip=xxxxx r-supplier-ip=xxxxx rs-time-taken=171 rs-server=xxxxx s-connect-type=Direct s-icap-status=ICAP_NO_MODIFICATION s-sitename=https.forward-proxy s-source-port=47952 s-supplier-country="None" sc-Content-Encoding=- sr-Accept-Encoding=gzip,%20deflate,%20identity x-auth-credential-type=- x-cookie-date=Wed,%2026-May-21%2006:33:08%20GMT x-cs-cer
12:03:08.920445 IP xxxxx > xxxxx.shell: Flags [.], seq 13271320:13278040, ack 1, win 4116, options [nop,nop,TS val 1363866026 ecr 1359753566], length 6720
E..t.Q..>..;
...
..s.S.......S.......].....
QJ..Q.1^tificate-subject=- x-cs-connection-negotiated-cipher=ECDHE-RSA-AES128-GCM-SHA256 x-cs-connection-negotiated-cipher-size=128 x-cs-connection-negotiated-ssl-version=TLSv1.2 x-cs-ocsp-error=- x-cs-Referer-uri=xxxxx x-cs-Referer-uri-address=xxxxx x-cs-Referer-uri-extension=- x-cs-Referer-uri-host=xxxxx x-cs-Referer-uri-hostname=xxxxx x-cs-Referer-uri-path=xxxxx x-cs-Referer-uri-pathquery=xxxxx x-cs-Referer-uri-port=443 x-cs-Referer-uri-query=- x-cs-Referer-uri-scheme=https...

When you open the packet capture and do a 'Follow TCP Stream' on the related stream, you will see that it does not contains the unknown characters. The 'Follow TCP Stream' window will have the following entries without the unknown characters.

<111>1 2021-05-26T06:33:08 xxxxx bluecoat - splunk_format - c-ip=xxxxx rs-Content-Type="text/html" cs-auth-groups=- cs-bytes=926 cs-categories="Finance" cs-host=xxxxx cs-ip=xxxxx cs-method=POST cs-uri-port=443 cs-uri-scheme=https cs-User-Agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" cs-username=xxxxx dnslookup-time=0 duration=0 rs-status=200 rs-version=HTTP/1.1 s-action=TCP_NC_MISS s-ip=xxxxx service.name=xxxxx service.group="Standard" s-supplier-ip=xxxxx s-supplier-name=xxxxx sc-bytes=15759 sc-filter-result=OBSERVED sc-status=200 time-taken=297 x-exception-id=- x-virus-id=- c-url=xxxxx cs-Referer=xxxxx c-cpu=- connect-time=76 cs-auth-groups=- cs-headerlength=918 cs-threat-risk=2 r-ip=xxxxx r-supplier-ip=xxxxx rs-time-taken=171 rs-server=xxxxx s-connect-type=Direct s-icap-status=ICAP_NO_MODIFICATION s-sitename=https.forward-proxy s-source-port=47952 s-supplier-country="None" sc-Content-Encoding=- sr-Accept-Encoding=gzip,%20deflate,%20identity x-auth-credential-type=- x-cookie-date=Wed,%2026-May-21%2006:33:08%20GMT x-cs-certificate-subject=- x-cs-connection-negotiated-cipher=ECDHE-RSA-AES128-GCM-SHA256 x-cs-connection-negotiated-cipher-size=128 x-cs-connection-negotiated-ssl-version=TLSv1.2 x-cs-ocsp-error=- x-cs-Referer-uri=xxxxx x-cs-Referer-uri-address=xxxxx x-cs-Referer-uri-extension=- x-cs-Referer-uri-host=xxxxx x-cs-Referer-uri-hostname=xxxxx x-cs-Referer-uri-path=xxxxx x-cs-Referer-uri-pathquery=xxxxx x-cs-Referer-uri-port=443 x-cs-Referer-uri-query=- x-cs-Referer-uri-scheme=https..

 

The proxy did send those characters but it is expected for the proxy to send those characters because it is part of TCP/IP packet been sent by the proxy, those part is actually the TCP and IP headers. For example, below is some of those characters:

QJ..Q.1^

Then if you look at the pcap below, taken from the proxy, the same characters is actually one of the TCP options.

 

 

 

Resolution

This is not an issue with the proxy but more an issue with the intermediate device or the Splunk server itself.

Attachments