Building your client application in Data Feed API

Products

• Email Security.cloud

Issue/Introduction

The ways you intend to use the data available through the Data Feeds API, along with your own preferences concerning programming languages and environments, determine the characteristics of the client application that you build. If simply accessing feed data is your goal, then using a script like NdfScript.py may meet your needs.

If you intend to sort or otherwise process data from the feeds as a prerequisite for importing the data to a SIEM or other system, then you may need to build a more complex application. You can extend the functionality of NdfScript.py itself, or write additional scripts in Python. You can also use NdfScript.py and NdfConfig.json as examples to speed the process of building a client in your preferred language or environment.

Environment

Data Feeds API

Email security cloud

Resolution

Your client application must meet the following minimum requirements:

• Support the storage of Email Security.cloud credentials that are configured initially by your Email Security.cloud portal
administrator.
• Specify the desired feed URI (all, malware, Threat isolation, click-time, spam, delivery, ec_reports, or test).
• Support the storage of persistent cookies that are received from the Data Feeds API's HTTP response(s). If the same
client accesses multiple feeds, cookies must be stored separately so that they are not inadvertently overwritten.
The client application must accept, store, and include stored cookies in requests as specified in RFC 6265, so as to
maintain proper session state and avoid duplicating or missing data.
• Support the storage of log data for monitoring and debugging.
• Support sending an HTTP GET request to 'reset' to obtain the initial cookie and to bootstrap the feed. For subsequent
requests, the client must accept and pass back persistent cookies to the service so that its place in the feed updates
properly.

The reset request is sent as a string representing the date from which to start reading the metadata with the format YYYY-MM-ddThh:mm:ssZ. When you call the service for the first time, if you don’t provide the reset request a 416 status code is returned. The reset request itself returns a 200 status code, but no data items. Requests that follow the initial reset request return the latest data. For example, using the URI https://datafeeds.emailsecurity.symantec.com/all?reset=2016-09-01T00:00:00Z  would reset the cursor for the all feed to midnight on the first of September 2016 (UTC). You must run the Python script again to return data from that point onward.

The Python script must be called repeatedly to get the most recent data, using a utility such as Cron (for UNIX-like environments) or Task Scheduler (for Windows). We recommend calling the script once per minute for organizations of 20,000 seats or more, and once every 5 minutes for smaller organizations. To consume the data that the script returns, point your SIEM to the directory that contains the JSON files. Your client application should clean this directory periodically to avoid the accumulation of previously consumed JSON files.