SyncAPI downloaded logs missing fields that are available in Portal downloaded logs

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

When downloading Cloud SWG logs via the SyncAPI, certain log request fields are missing when checking log entries on our SIEM.

Downloading the HTTP logs from the Portal does show all the required/expected fields.

Examples of missing fields from SyncAPI downloads include the WSS Agent host RFC1918 IP address (x-client-agent-ip), or the dedicated IP address application name (x-symc-dei-app).

Environment

Cloud SWG.

SyncAPI.

Splunk SIEM.

Cause

By default, newly added HTTP logging fields are not added to the default download to avoid parsing risks on the SIEM.

Resolution

Use the field parameter available with SyncAPI to manually add the fields that are needed by SIEM.

In the specific example above, one would add the following two highlighted parameters at the end of the SyncAPI request to obtain the WSS Agent host IP address and dedicated IP application name - the other parameters are the default parameters sent back with every request logged:

curl -s -o test.zip -H "X-APIUsername: <username>" -H "X-APIPassword: <password>" \
    "https://portal.broadcom.com/reportpod/logs/sync?startdate=0&enddate=0&token=none&fields=x-bluecoat-request-tenant-id,date,time,x-bluecoat-appliance-name,time-taken,c-ip,cs-userdn,cs-auth-groups,x-exception-id,sc-filter-result,cs-categories,cs(Referer),sc-status,s-action,cs-method,rs(Content-Type),cs-uri-scheme,cs-host,cs-uri-port,cs-uri-path,cs-uri-query,cs-uri-extension,cs(User-Agent),s-ip,sc-bytes,cs-bytes,x-icap-reqmod-header(X-ICAP-Metadata),x-icap-respmod-header(X-ICAP-Metadata),x-data-leak-detected,x-virus-id,x-bluecoat-location-id,x-bluecoat-location-name,x-bluecoat-access-type,x-bluecoat-application-name,x-bluecoat-application-operation,r-ip,r-supplier-country,x-rs-certificate-validate-status,x-rs-certificate-observed-errors,x-cs-ocsp-error,x-rs-ocsp-error,x-rs-connection-negotiated-ssl-version,x-rs-connection-negotiated-cipher,x-rs-connection-negotiated-cipher-size,x-rs-certificate-hostname,x-rs-certificate-hostname-categories,x-cs-connection-negotiated-ssl-version,x-cs-connection-negotiated-cipher,x-cs-connection-negotiated-cipher-size,x-cs-certificate-subject,cs-icap-status,cs-icap-error-details,rs-icap-status,rs-icap-error-details,s-supplier-ip,s-supplier-country,s-supplier-failures,x-cs-client-ip-country,cs-threat-risk,x-rs-certificate-hostname-threat-risk,x-client-agent-type,x-client-os,x-client-agent-sw,x-client-device-id,x-client-device-name,x-client-device-type,x-client-security-posture-details,x-client-security-posture-risk-score,x-bluecoat-reference-id,x-sc-connection-issuer-keyring,x-sc-connection-issuer-keyring-alias,x-cloud-rs,x-bluecoat-placeholder,cs(X-Requested-With),x-random-ipv6,x-bluecoat-transaction-uuid,x-symc-dei-app,x-client-agent-ip"