Minimizing possible data inconsistency when upgrading an Encryption Management Server cluster

Products

• Encryption Management Server
• Gateway Email Encryption

Issue/Introduction

Most organizations who are upgrading an Encryption Management Server cluster in a production environment wish to avoid any downtime.

The natural approach is to point all endpoints to one cluster member while the first of the remaining cluster members is upgraded and then point all endpoints to the first upgraded server while the remaining ones are upgraded.

However, this approach does risk the possibility of data inconsistencies occurring. This is because data replication does not occur between cluster members unless the cluster members are running the same release of Encryption Management Server.

This issue is particularly relevant when undertaking an upgrade to a major release where you need to backup the system, install from ISO media and restore the backup. This is simply because the upgrade takes far longer than a "PUP" or minor upgrade. For example, when upgrading to release 10.5 or above from 10.4.2 MP5 or below.

Consider the scenario where there are 2 servers in a cluster, server1 and server2:

1. DNS or a load balancer is configured so that the Encryption Desktop clients all point to server1. Likewise, if the Encryption Management Servers process email, all SMTP traffic routes through server1.
2. You upgrade server2. This involves taking a backup of server2, installing from ISO and restoring the backup. This will take several hours at a minimum.
3. While server2 is being upgraded, Encryption Desktop users could still enroll or re-enroll to server1. In the case of new enrollments, new user and computer accounts would be created on server1. In terms of SMTP traffic, if an internal user sends an email that gets routed through Encryption Management Server, an account for that internal user will be created on server1 if an account does not already exist. If the Encryption Management Server has the Web Email Protection service enabled, external user accounts, mailboxes and messages stored in those mailboxes may also be created.
4. Once server2 is upgraded, any new accounts, etc on server1 will not get replicated to server2 because replication does not work if the servers are on different releases.
5. DNS or a load balancer is updated to point all Encryption Desktop clients to the upgraded server2. Any new accounts, etc created on server1 while server2 was being upgraded will not exist on server2.
6. You upgrade server1.
7. Both servers are now on the same release so real time replication will start working.
8. After "incremental" replication has run overnight, the servers will be in sync.
9. DNS or a load balancer can now be configured to point to server1.

At step 5 you may have a data inconsistency issue which will only be resolved after step 6.

Environment

Symantec Encryption Management Server 10.5 and above.

Resolution

Clearly one way of avoiding this potential data inconsistency is to stop all traffic reaching any cluster member until at least one cluster member is upgraded. In other words, a period of downtime.

While this is the safest option, it may not be desirable or practical. Therefore, consider doing the following while the first cluster member is being upgraded in order to mitigate the issue:

1. If Encryption Management Server is configured to use Directory Synchronization whereby it points to one or more Active Directory domain controllers, disable Directory Synchronization.
2. If the Web Email Protection service is enabled, pause it.

If Directory Synchronization is usually enabled, disabling it will do the following:

1. New Encryption Desktop users will not be able to enroll.
2. Existing Encryption Desktop users will not be able to re-enroll.
3. When an internal user who does not already have an account on Encryption Management Server sends an outbound email through it, Encryption Management Server will not be able to create a new account for the user.
4. Encryption Desktop users will still be able to download policy, upload logs and lookup keys on Encryption Management Server.
5. Email sent by users who already have an account on Encryption Management Server will still be sent.
6. Email sent by existing users to new external users will still result in new Web Email Protection or PDF Email Protection accounts being created.
7. Existing Web Email Protection or PDF Email Protection users will be able to work normally.
8. Inbound email will be processed as normal.

To disable Directory Synchronization, in the administration console, navigate to Consumers / Directory Synchronization and click on the Disable button. It will then appear like this. Click on the Enable button to enable it again:

Once the first cluster member has been upgraded and all endpoints are pointed to it, Directory Synchronization can be enabled on it.

Pausing the Web Email Protection service will do the following:

1. External users will not be able to login at the portal.
2. They will therefore be unable to read or send any email or, in the case of new users, set their password.
3. Email sent by users who already have an account on Encryption Management Server will still be sent.
4. Email sent by existing users to new external users will still result in new Web Email Protection or PDF Email Protection accounts being created.
5. Inbound email will be processed as normal.

To pause the Web Email Protection service, in the administration console, navigate to Services / Web Email Protection and click on the Pause button. It will then appear like this. Click on the Resume button to resume it:

Once the first cluster member has been upgraded and all SMTP traffic is pointed to it, its Web Email Protection service can be resumed.

As you can see, these mitigation measures do not completely prevent any potential data inconsistency issues but they do minimize them.