Clustering and Data Replication Stops Working Between SEMS Servers After Upgrade from 3.3.2 to 3.4 and Later

Products

• Encryption Management Server

Issue/Introduction

Members of a cluster, after an upgrade of SEMS 3.3.2 to 3.4 and later, may have their replication process stop working. Cluster communication could be completely broken, or some cluster members could see others as active while others are unable to communicate. You can check this in the SEMS web console under System > Clustering and checking the Replication Status from each servers' web consoles.

Multiple scenarios can cause this.

• Replication Status shows as Down on the affected server from other members of the cluster. The affected server shows other servers as active.
  "Cannot bind port [IP.Address]:389 [or 636], the binding already exists."
  There could be other similar port binding messages in the clustering logs on the affected server.
• When trying to restart stunnel from command line: "file /etc/stunnel/stunnel.conf line 30: Bad verify level"

Environment

Two or more Encryption Management Servers in a cluster
Upgraded to 3.4.x from a previous version.

Cause

STunnel service is not running. STunnel is responsible for delivering the replication information to the other cluster members securely.

Resolution

The STunnel service will not run if there are issues with the /etc/stunnel/stunnel.conf file. The upgrade process might place duplicate Keyserver access entries into Keys > Keyservers. Remove all duplicates, and refresh the page to be certain they have not repopulated. If the error relates to the bad verification level, edit the stunnel.conf file and make sure "verify = 1" for the entry.

Stunnel can be restarted with the command "services stunnel restart". Restart the services or reboot the server once stunnel is corrected, and check status of replication. It should begin to replicate automatically.