Manually Synchronize a Multi-node Multiwrite with Disp Recovery Implementation

Products

• CA Directory

Issue/Introduction

By design, a multiwrite with disp recovery is capable of keeping a multi-node implementation data in sync. However, under cetain extreme coniditions, there may still chance when you need to perform a manual synchronization to keep all the data synchronized. 

Environment

Complement: CA Directory 14.1

Resolution

The Assumptions:

1. All the servers' system clock are synced up using the same time server. For Windows server implementation, please also make sure that the windows-high-resolution-time is set to true. Further discussions with regard to Windows server implementation, please see the following product documentation links:

• DSA Rejects a Delete Request That Closely Follows a Modify or Create Request
• set windows-high-resolution-time Command -- Change the time resolution for Windows

2. Three servers or more are configured in a multiwrite with disp recovery and the transaction logs are enabled with flushing 

Assume the servers named as A, B, C, ..., etc are configured as multi-write with disp recovery, using on the following two settings

dsa-flags     = multi-write, no-service-while-recovering
set multi-write-disp-recovery = true;

In addition, the transaction logs are enabled with flushing through the effective settings of the two:

set disable-transaction-log = false;
set disable-transaction-log-flush = false;

3. Assuming the data on Server A will be used to sync up all other servers.

The Suggested Manual Synchronization Steps

Please refer to the "Multiwrite Group Peer Recovery" section of the following product documentation link:

Topology Sample and Disaster Recovery

1. Start with synchronizing the server B: go to server B to stop the DSA by running the following on server B

   dxserver stop ClusterNodeB

2. Go to the other servers, namely A, C, ..., etc, to set the disp time before the taking the onlinebackup by running the following command on all three servers:
   

   dxdisp ClusterNodeB

3. Go to server A immediately after all the dxdisp commands executed to take an online backup by running the following:
   

   dxserver onlinebackup ClusterNodeA

   and wait for the onlinebackup to finish by monitoring the warn log on server A to see when the dump has started, and equally importantly completed. When completed, there will be a ClusterNodeA.zdb file under the data subdirectory we need. Note that this command is equivalent to the telnet to console method documented in the above documentation link
   
   
4. Go to server B to recover ClusterNodeB immediately after the ClusterNodeA.zdb is ready by doing the following:
   
   4.1 delete the ClusterNodeB.db and ClusterNodeB.tx under the data subdirectory
   4.2 copy the ClusterNodeA.zdb as ClusterNodeB.db under the data subdirectory
   4.3 run the 'dxserver start ClusterNodeB'
   4.4 wait for the progress of the recovery of server B to complete by following in the alarm-log for the ClusterNodeB on server B.
   
   
5. After the server B has been recovered, repeat the same steps for the rest of the servers, namely server C and others Please keep in mind that you need to redo the server A onlinebackup and the dxdisp command for each server you are recovering. Once all servers B, C, ..., etc are done, the whole process is done.