HTTP logs downloaded via SyncAPI missing fields available from the Portal download HTTP logs.

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

SIEM downloading Cloud SWG logs from SyncAPI endpoints.

Cloud SWG Admin cannot find all HTTP log fields that SOC team requires e.g. the x-client-agent-ip that reports the WSS Agents host IP as opposed to egress IP address; or the dedicated IP address / Application name fields.

Missing SynAPI log field entries are available from the Portal log fields when exporting HTTP logs from there.

How can I make sure that my SIEM can download all the fields I need via the SyncAPI?

Environment

Cloud SWG.

SyncAPI.

Cause

Do not want to change the format of the SyncAPI downloaded logs, which could break SIEM ingestion or data rendering process.

Resolution

The SyncAPI provides a new 'field' option to append missing HTTP log fields that are not sent by default.

The example below shows how this fields parameter can be used to pull the dedicated IP address and WSS Agent host IP address field values referenced above - NOTE that the X-APIUsername and X-APIPassword must be changed to match the credentials defined on the Cloud SWG Portal.

curl -vvv -s -o WSS_Logs.zip -H "X-APIUsername:xxxxx" -H "X-APIPassword:yyyyyy" "https://portal.threatpulse.com/reportpod/logs/sync?startdate=1678181608000&enddate=0&token=none&fields=x-bluecoat-request-tenant-id,date,time,x-bluecoat-appliance-name,time-taken,c-ip,cs-userdn,cs-auth-groups,x-exception-id,sc-filter-result,cs-categories,cs(Referer),sc-status,s-action,cs-method,rs(Content-Type),cs-uri-scheme,cs-host,cs-uri-port,cs-uri-path,cs-uri-query,cs-uri-extension,cs(User-Agent),s-ip,sc-bytes,cs-bytes,x-icap-reqmod-header(X-ICAP-Metadata),x-icap-respmod-header(X-ICAP-Metadata),x-data-leak-detected,x-virus-id,x-bluecoat-location-id,x-bluecoat-location-name,x-bluecoat-access-type,x-bluecoat-application-name,x-bluecoat-application-operation,r-ip,r-supplier-country,x-rs-certificate-validate-status,x-rs-certificate-observed-errors,x-cs-ocsp-error,x-rs-ocsp-error,x-rs-connection-negotiated-ssl-version,x-rs-connection-negotiated-cipher,x-rs-connection-negotiated-cipher-size,x-rs-certificate-hostname,x-rs-certificate-hostname-categories,x-cs-connection-negotiated-ssl-version,x-cs-connection-negotiated-cipher,x-cs-connection-negotiated-cipher-size,x-cs-certificate-subject,cs-icap-status,cs-icap-error-details,rs-icap-status,rs-icap-error-details,s-supplier-ip,s-supplier-country,s-supplier-failures,x-cs-client-ip-country,cs-threat-risk,x-rs-certificate-hostname-threat-risk,x-client-agent-type,x-client-os,x-client-agent-sw,x-client-device-id,x-client-device-name,x-client-device-type,x-client-security-posture-details,x-client-security-posture-risk-score,x-bluecoat-reference-id,x-sc-connection-issuer-keyring,x-sc-connection-issuer-keyring-alias,x-cloud-rs,x-bluecoat-placeholder,cs(X-Requested-With),x-random-ipv6,x-bluecoat-transaction-uuid,x-symc-dei-app,x-symc-dei-via,x-client-agent-ip"