Load balancing DLP16.0+ OCR servers

Products

• Data Loss Prevention Core Package

Issue/Introduction

DLP 16.0 load balancing methods have changed to HTTP2 L7 load balancing.

 

If your load balancers are still configured for L4 load balancing in the Enforce UI you may see an error 4800 OCR service is busy.

And you may see where 1 OCR server gets a constant stream of data, and others remain idle.

Environment

DLP 16.0 and newer

Cause

By popular demand in DLP 16.0 OCR L4 load balancing has been deprecated, and it has been replaced with L7 application load balancing for increased performance.

Now rather than opening a new connection for each image, the connection is opened to the load balancer and remains open. This increases performance by not requiring a new TCP/IP handshake in between each image. And the newer L7 application load balancer is able to route the load based on the server availability rather than just a round robin. 

Resolution

For best results you should enable Layer 7 application load balancing, for help with exchanging certificates between the load balancer and the OCR servers please see:

Exporting Private Keys, Certificates, and Trusted Certificates from a 15.x OCR Server

The configuration of the load balancer should be the same as any other HTTP2 application load balancer.

 

If enabling L7 load balancing is not an option, you can use DNS load balancing for very limited usage.

Please note: While this is acceptable for a one to many configuration it is not suitable for a many to many configuration.

So if there is 1 detection server for several OCR servers this configuration while not ideal is acceptable. 

This will not work well if you have several detection servers sharing OCR servers. 

Once DNS load balancing is enabled, the detection server will receive multiple addresses and will round robin the received addresses, if multiple detection servers are used there will be nothing to prevent a over/under utilization of one OCR server or another. This is why L7 application load balancing is preferred where the load balancer should intelligently determine which OCR server has the least load and send the data accordingly. 

In order to setup DNS load balancing you simply create a singular FQDN, "AKA: OCRcluster" and then create multiple A records for each OCR server using the same FQDN, with the unique IP address of each server.

You then add that FQDN to your OCR Engine Configuration.

Now when the detection server pulls down the FQDN it will receive multiple IP addresses and will round robin between them.

 

Additional Information

Sample configuration for ngenix load balancer is included below.

Instructions for F5 can be found on their site at:

https://my.f5.com/manage/s/article/K08041451

For issues with configuration of F5 load balancers please see the attached file BIG-IP_Service_Provider__Generic_Message_Administration.pdf.

In conversations with F5 they shared the attached pdf with us, stating it can be helpful to implement F5 to perform RPC request level load balancing as opposed to the connection level one that is generally used. The F5 engineer we met with mentioned with near certainty that it is possible to configure F5 this way, but that it is not straightforward to do. So it is recommended to contact F5 support for assistance in this configuration.

 - If customers find the client side load balancing acceptable, and do not want to pursue making F5 work with OCR, the client side load balancing option "DNS Load Balancing" as indicated in this KB is fully supported, and very effective operating much the same way the old conventional load balancing method did. "AKA Round Robbin"