Defining / Specifying Partition on ProxySG / EdgeSWG when using kafka client to upload access logs

Products

• ISG Proxy
• ProxySG Software - SGOS

Issue/Introduction

You have configured ProxySG to upload access logs to a kafka client/receiver and have defined the 'topic name' in the upload configuration. You also want to define the partition / partition name of the topic to which you want the logs uploaded.

Environment

ISG Proxy 7.x

Resolution

 In the ProxySG (SGOS) logging configuration for Kafka, you will find the field to define the Topic Name, but there is no field to explicitly define a Partition. The ProxySG will automatically distribute log messages across the available partitions of the specified topic using its internal Kafka producer logic. This is not customizable.

 

Additional Information

To ensure that consumers receive log data in the correct chronological order when messages are distributed across multiple partitions, you must utilize the timestamp field present within the log entries for post-consumption sorting. You will have to use Consumer-Side Sorting, log management and SIEM platforms (like Splunk, Elastic, etc..) are designed to handle this behavior. They ingest data from all partitions in parallel and then use the event timestamp (generated by the ProxySG at the time of the transaction) to build a chronologically ordered view of the events. 

Alternatively, you can use Single Partition Configuration (If Strict Real-Time Order is Required). If your application requires logs to arrive at the consumer in strict sequential order without secondary sorting, you should configure the Kafka topic on the broker to have only one partition. Note, This significantly limits the throughput and scalability of the topic, as all log data must be processed by a single consumer thread and is not recommended.

Please note, Reliable sequencing is always achieved by sorting based on the time and date fields within the access log format. For timestamps to be reliable across multiple ProxySG appliances or within a complex Kafka environment, ensure all appliances are synchronized using a reliable NTP server.