Using DNS Round Robin and Load Balancers, Front-End Security Applications and Reverse Proxies with PGP Encryption Server
Article ID: 156803
Updated On:
Products
Issue/Introduction
Load Balancers are a valuable and critical part of failover in any network infrastructure and using a Load Balancer with the PGP Encryption Server (Symantec Encryption Management Server) is no exception. When done properly, this can offer the failover that is needed in order to continue to have clients communicate with the server when one server may not be available.
When no load balancer is used, but there is a PGP Encryption Server cluster configuration, the PGP Encryption Desktop (Symantec Encryption Desktop) will enroll to a specific server. If that server is unavailable, then the clients will no longer be able to download policy or upload logging data to the server, even though there is a cluster.
Symantec Web Email Protection (WEP) users may not see all their new messages when they log in if one server is not available, even though other nodes are working, so Load Balancers offer a critical part of resilience and availability in a network architecture.
This article is not intended to be a walkthrough for how to configure a Load Balancer, but will offer some guidance on what you can do to properly configure the Load Balancer so that if one of the PGP Encryption Server nodes is unavailable, proper failover can occur.
For information on Best Practices and Recovery for the SEE Management Server, see the following article:
Cause
By default, the PGP Encryption Desktop installation file downloaded from PGP Encryption Server contains the FQDN (fully qualified domain name) of the server from which the installation file is downloaded.
During the enrollment process, the client is bound to a single PGP Encryption Server (SEMS). If that server is unavailable, the client will not be able to connect.
For Web Email Protection, a user may be directed to a cluster member that has not received the replicated data yet.
Resolution
Table of Contents
1. Reverse Proxies & Front-End Security Applications
Reverse Proxies should be fine as long as the internal TLS packets are not modified. Symantec Encryption products all use TLS, which is a standard communications protocol and as long as the Reverse Proxy does not change the actual packet content, this should be fine to use in the environment for communications from the Clients to the servers.
As long as the client-to-server communications are simply passed through the reverse proxy, this should be supported.
Front-End Security Applications can also be used if you would like to filter http traffic going to the PGP Server, so long as the data packets are *not* modified.
In addition to packets not being modified, and much like a Load Balancer, the sessions to PGP server must be kept persistent.
For Example: If a user logs in to their Web Email Protection account, and the session goes through a front-end security application (FESA), this same FESA node must stay persistent and not change the session to a different FESA node such as the following flow:
1. User presented to the login page (In the background, FESA1 is used).
2. User enters their credentials (In the background, FESA1 is still used).
3. User is logged in to the account and clicks on an element in their WEP inbox, such as "Compose" (In the background, FESA1 is still used).
4. User sends the email composed (FESA1 is still used).
5. User Logs out (FESA1 is still used for logout event).
6. User goes back to the WEP login page. In the Background, FESA2 may then be used as this is a completely different session.
Each WEP login is protected with unique CSRF tokens and these tokens must not be modified, or the session will quit.
2. Considerations for "Health Checks" for Load Balancers and SMTP
It is common that a Load Balancer will do an SMTP "Health Check" for the PGP server to ensure that not only is the appropriate port listing, but also a "220 — SMTP Service ready." status is available.
Because the PGP server behaves as a "proxy" server, if the Load Balancer is checking via telnet logic for SMTP, if the PGP server has a next hop, it must also be able to connect on that port, or the health check will fail even if the SMTP port is open and listing.
As an example, consider the outbound mail flow:
SMTP Server --> PGP Server --> MTA
If the PGP Server is not able to make an SMTP connection to the MTA above, then this Load Balancer health check may not succeed, reporting the PGP server is down, even though the service is up. Make sure the PGP server is also able to make a telnet connection to the MTA in this example for this to succeed.
Load Balancers and DNS Round Robin
In a clustered environment, an FQDN that can resolve to more than one PGP Encryption Server is required in order to provide fail over for PGP clients. If one server is not available, the LB will redirect to the other server that is available.
For this reason, before you start enrolling clients, establish a good name that the clients will connect to via the Load Balancer so that the Load Balancer will the redirect the traffic.
*This FQDN is specified in the PGP Encryption Server text box in the Consumers / Groups / Download client page of the admin console, such as keys.example.com.
For example, if you have keys1.example.com for keys Node 1, and keys2.example.com for node 2, then when you download the client, configure the name "keys.example.com" and that will be the host the LB will be using. When the SED client is installed, it will reach out to keys.example.com; the load balancer will accept the connection and it will then redirect to either keys1 or keys2.
Similarly, if keys1 and keys2 host Web Email Protection, then the Load Balancer could be configured with "keys.example.com" and this is the host DNS will resolve to, and the Load Balancer will then direct traffic to keys1 or keys2 depending on the situation.
Because the clients connect to the server over a secure channel, each PGP Encryption Server in the cluster requires an SSL certificate with a Common Name that matches the FQDN to which the clients connect. For example, the certificate Common Name might be keys.example.com.
Symantec recommends that cluster replication should be configured to use network Interface 1 and that each interface used for clustering should have a unique SSL certificate. See article 154069 for further information. For example, in a three member cluster consisting of keys1.example.com, keys2.example.com and keys3.example.com, Interface 1 of keys1 would have a certificate with the Common Name keys1.example.com, Interface 1 of keys2 would have a certificate with the Common Name keys2.example.com and Interface 1 of keys3 would have a certificate with the Common Name keys3.example.com.
Since Interface 1 is used for replication and each server has a unique certificate assigned to Interface 1, client communications need to be handled by a different Interface, usually Interface 2.. The same SSL certificate needs to be assigned to all servers that handle client traffic because the clients need to trust it. For example, all servers might have the certificate for keys.example.com assigned to Interface 2.
Important Note 1: If Load Balancers are not configured properly, this can cause unexpected behavior on PGP Server, including Unhandled Exceptions. This is due to a user clicking on one link in their WEP inbox, and the LB redirects to a different server midstream. Symantec Enterprise Division recommends having PGP Encryption Server be configured for only one node. If the node is unavailable, then redirect to other servers, but generally have only one PGP Encryption Server be configured in the rotation.
In all cases, the so-called "sticky bit" or "persistence" should be set so that clients do not connect to a different server during their session. There are several methods to use persistence and your Load Balancer may have more or less of these features:
*Cookie Persistence - The Load Balancer provides a cookie value in the connection stream and when the client uses this, the LB will always be directed to a specific PGP Encryption Server. This may or may not work in some environments so if cookie persistence is causing issues, look for a different method for persistence.
Important Note 2: SED/PGP Encryption Server 10.5 may not support cookie-based persistence so be sure to test this before going to production. Symantec Encryption Desktop uses this directive for its enrollment and will expect only one "set-cookie" value. If a Load Balancer inserts this value as well, this can cause enrollment to fail.
(EPG-24107/EPG-24286 - Starting with 10.5.1, this set-cookie value is set to "ovidCookie" and adding an exception on the Load Balancer to get enrollment working should no longer be necessary - Contact Support for more information on this if you continue to run into issues).
*Source IP Persistence - The Load Balancer will review the source IP that connected to the PGP Encryption Server and based on that source, it will always keep that source going to the same server.
*Duration based Persistence - This may keep the connection going to only one host for a certain amount of time and will not switch that connection for any other reason.
Consult your Load Balancer documentation and configuration options to choose which is best for you.
___________________________________________________________________________________________________
3. Creating the Symantec Encryption Desktop Client from PGP Encryption Server
Note for Load Balancers - TLS Passthrough VS TLS Renegotiation, Wildcard Certificate VS Single FQDN Certificate:
If you are using a Load Balancer to route communications to the PGP server, enter the FQDN the Load Balancer will be using. For example, if you have two PGP Servers, one called "pgp01.example.com" and the other "pgp02.example.com", you'll want to use a name that will resolve to the Load Balancer and then Load Balancer will then redirect traffic to one of the two servers in question.
In one example, we could call the Load Balancer FQDN "keys.example.com". So when you build the PGP client, enter this hostname and this will assign "keys.example.com" for the installer package, this is called the "PGPSTAMP", which is the FQDN for the PGP Server. Post install, when the PGP client attempts to check in, it will attempt to resolve the PGPSTAMP value, or "keys.example.com" FQDN, which will go to the Load Balancer.
At this point, you want to think about how the TLS communications should behave.
Because the PGPSTAMP is pointing to "keys.example.com", the TLS certificate being used for the interface should also match "keys.example.com", whether it is the PGP Server that presents the TLS certificate or the Load Balancer.
TLS Passthrough
If you are using the "passthrough" method on the Load Balancer, meaning the Load Balancer will not be presenting any certificates for the TLS connection and is simply sending to one of the two PGP Servers, ensure the certificate that is being used matches the same FQDN used to create the client.
If the TLS certificate is created for "keys.example.com", but the servers are called "pgp01" and "pgp02", then the PGP client is going to pop up a certificate warning. To avoid this scenario, you can use a wild card certificate so that regardless of the hostname of the PGP server, as long as the domain is the same, the certificate warning will not pop up. For example, the hostname "pgp01.example.com" and "pgp02.example.com" will not pop a certificate warning if a wildcard certificate for ":*.example.com" was created.
Optionally, if a wildcard certificate is not possible, you can try adding SANS Alternative values for each PGP servers assigned.
TLS Renegotiation
If a wildcard certificate is not being used and SANS Alternative Values are not available, then Symantec recommends creating the certificate for "keys.example.com" and assigning that certificate to the Load Balancer and have the Load Balancer renegotiate. All the PGP clients will be connecting with TLS via the Load Balancer, and the Load Balancer will renegotiate to the PGP servers. The PGP clients will not be aware of the TLS connection being made from the Load Balancer to the PGP servers if TLS renegotiation/TLS Termination is being done on the Load Balancer and should avoid any TLS pop ups.
Caution: If you are using an Internal CA, ensure the Root and Intermediate Certificates are uploaded and fully trusted into the Trusted Keys section on the PGP Encryption Server. In some cases using an internal CA, you may need to specifically assign the server certificate to be trusted. In other words, some environments will not accept any certificates unless they were specifically added, even if they are signed by a trusted Internal CA so it's a good idea to test this thoroughly before deploying to the entire organization
There may be other network appliances similar to what Load Balancers do. As this is all standard TLS, the above scenarios can apply to other network solutions for load balancing.
180244 - HOW TO: Download Encryption Desktop Client Installers from the PGP Encryption Server (Symantec Encryption Management Server)
___________________________________________________________________________________________________
Round Robin for load balancing is not supported. A single PGP Encryption Server can easily service tens of thousands of clients so the main advantage to having a cluster of PGP Servers is high availability and not load distribution.
DNS can be used to point the FQDN that PGP Encryption Desktop clients use to a specific PGP Encryption Server. If that server becomes unavailable, DNS can be manually updated to point the FQDN to another server in the cluster. For example, keys.example.com might point to Interface 2 of keys1 which has an IP of 192.168.0.50. If keys1 is unreachable, the DNS entry might be changed to point to Interface 2 of keys2 which has an IP of 192.168.0.75.
As an alternative to DNS, a load balancer can be used. However, the load balancer must not be configured to use Round Robin, even though it is the default for most load balancers. Depending on the specific load balancer, the following methods can be used in order of preference:
- Fail-over
- If you have two PGP Encryption Servers, configure the load balancer to direct all traffic to one server and fail over to the other if the first is unavailable. Nearly all load balancers can do this.
- Ordered List or First Alive
- This is where the servers are listed in a specific order. For example, with three servers it might be: 192.168.0.50, 192.168.0.75, 192.168.0.100.
- Clients will always be directed to the first host in the list unless it is unavailable, in which case clients will be directed to the second host and so on.
- This method is available with some load balancers from, for example, Cisco and IBM.
- Weighted Round Robin or Ratio
- This is where each host in the list is given a weighting so that traffic is not distributed evenly to each of them.
- The host with the highest weighting receives the most requests. For example, the load balancer vendor may allow weightings between 1 and 10 to be used. The server with a weighting of 10 will receive ten times more traffic than the server with a weighting of 1.
- By giving one server in the cluster the maximum weighting and the other servers the minimum weighting, an acceptable load distribution profile can be achieved. For example, 192.168.0.50 might be given a weighting of 10 and both 192.168.0.75 and 192.168.0.100 given a weighting of 1. This would mean that 192.168.0.50 received the vast majority of the traffic.
- This method is available with many load balancers from, for example, Cisco, F5, Citrix and IBM.
Additional Information
154069 - Best Practices for Symantec Encryption Management Server Clustering
269071 - Best Practices for Disaster Backup and Recovery with PGP Encryption Server (PGP)
ISFR-2129/EPG-27236 - Allow Multiple PGP servers to be configured on a single PGP client.
If you would like to be added to this request, please reach out to Symantec Encryption Support and provide this ID.
Feedback
