Parsing of Cloud SWG malware streamed logs failing after recent reporting update

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Event streaming enabled to AWS Cloud bucket for Cloud SWG proxy, CFS, audit and malware events.

SIEM ingesting logs from AWS Cloud bucket show all the expected logs.

After a documented Cloud SWG status page reporting update however, malware logs are no longer visible - proxy, audit and CFS events are.

Event streaming format for all events is JSON.

When it fails, a local parser retrieving streamed logs from AWS Cloud bucket seems to fail for the malware logs only.

Environment

Cloud SWG event streaming.

AWS Cloud bucket.

Cause

Local AWS lambda parser function hard coded to look for certain format, when JSON is not ordered.

Resolution

Update the parser code to avoid checking fields in specific orders.  Cloud SWG reporting service do not guarantee any field ordering for JSON.

With NDJSON, which is the format Cloud SWG reporting uses, field order or even the fields that exist shouldn't matter at all. Any changes to the JSON record fields will be documented should they happen (which is very infrequently, and usually). With NDJSON events are encoded as JSON documents and separated by newline characters; each line contains one event encoded as a separate JSON object. 

For admins that are not using JSON (ELFF for example), Broadcom recommends switching to JSON to avoid parser problems should anything change, which we potentially will as we add new things or for whatever reason. We don't even guarantee field order from message to message (much less release to release).