After upgrading to SMP 8.1 RU5 or later replication jobs suddenly take very long or do not complete due to database contention and deadlocking.

book

Article ID: 171849

calendar_today

Updated On:

Products

Management Platform (Formerly known as Notification Server) Software Management Solution

Issue/Introduction

The SMP logs are filled with warnings and errors similar to the following (excerpts)

Duplicate resources merge succeeded, resource(to): 'Timer.exe' (53b1379d-0d39-4074-bc36-6eaeb0029dd9), duplicate(from): 7cc254e9-2d03-4a22-a805-3bc8d8f94498, type: 018b191b-47ae-4180-9fcd-7f3cea4f1e12

Description: Hierarchy resource synchronization(merge) event[4ca7251b-eade-492e-b3e5-a122111e7c01] was queued for replicaiton in DOWN direction. ResourceTo:[ec7f5645-5870-44bd-983f-34d60cdc76fa]


Desciption: Operation not in progress. jobId:{a0a213c9-f27e-4002-bb0f-34129a1b9b6e}, operationId:{c31e88de-4366-4fd7-bcd3-0cf9f69c97a4}, stage:Wait, type:Export, itemType:Item, status:Completed


Description: Operation was blocked by merge activity for: 15718.9094 ms.
   at Altiris.NS.ResourceManagement.ResourceAssociationImporter.MapMerges(IEnumerable`1 operations, Int32 count)
   at Altiris.NS.ResourceManagement.ResourceAssociationImporter.LoadResourceAssociationsImpl(ResourceAssociationDataCollection resourceAssociationDataCollection)
   at Altiris.Database.DatabaseContext`1.RetryAction(Int32 retries, Boolean transacted, Getter`1 getContext, Action`1 action, Action`1 retry, DeadlockRetryArgs& re, Exception& exception, Boolean inTransaction)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetryHelper(Int32 retries, Boolean inTransaction, Getter`1 getContext, Action`1 action, Action`1 retry, Boolean transacted)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetry(Int32 retries, Boolean startNewTransaction, IsolationLevel isolationLevel, Boolean independentContext, Action`1 action, Action`1 retry)

   
 

The resource types being merged are almost all "File018b191b-47ae-4180-9fcd-7f3cea4f1e12.

Cause

The replication Rule "File Resources for executable files" was modified in the release to replicate ALL discovered file types. This causes each child to replicate every file type they have discovered (via Application Metering and Inventory) to the parent. The parent then begins sending synchronization events out to all child servers which causes them to merge the GUID the parent assigned to the same executable object with the GUID currently assigned to it in their own database. Ownership is also changed.

The mass of synchronization  /merge events causes major SQL deadlocking and slowdown.


Not all executable file objects need be replicate up from the child servers. Only the one being monitored by Application Metering.

 

Environment

8.1 RU5 and greater

Resolution

The severity of the problem is weighted by the number of servers in the Hierarchy. If there is only one parent and one child it may be best to just wait until they are finished.

If there are multiple children-- the best way to get control of the situation is to do the following.

  • Open the parent server's SMP console and go "Settings > Notification Server > Hierarchy".  Right click on the parent object and select "Disable Replication"
  • Follow the same step by opening the Child servers' console and disabling replication.
  • In the SMP console, in the Hierarchy section, select the "Jobs Management" tool near the bottom in the left pane. Look for all active jobs and selec the red X tool in the tool bar to stop each job. Do this on all servers.
  • Disable the "File Resources for executable files" replication rule.  This rule would normally need to be disabled on the parent and replicated down, but in this case run the following SQL statement against the database of each server in the Hierarchy.      

UPDATE ItemActive set Enabled = 0 where Guid = '38A24D5A-5E2C-41FD-AE21-D5360D5B7D59'
UPDATE ReplicationJob set State = 6 where State < 4

  • Stop IIS:    IISRESET -stop
  • Stop the Service "AltirisSupportService" and "Altiris Service"
  • On each server open Windows File Explorer and go to "C:\ProgramData\Symantec\SMP\Replication\Temp"  Delete any folders and files in the Temp folder.
  • Start IIS (IISRESET -start) and the service "AltirisSupportService"
  • Open the console on the parent and subsequent children and enable replication again.
  • At this point you can manually start a Differential replicaiton between parent and a child or wait for the normal schedules to run.  
  • If there is a need to replicate EXE files for the purpose of metering their usage etc then follow the steps below.

 

The following steps can be used to replicate only desired executables instead of all

  1. Make a clone of filter "Inventoried Executable Files (Up)"
    1. Go to Manage > Filters
    2. In the filter tree, go to Filters > Software Filters > Agent and Plug-in Filters
    3. Right-click filter "Inventoried Executable Files (Up)" and select "Clone".
    4. Name it "Custom Filter – Only Required Inventoried Executable Files (Up)"
    5. Find newly created filter in the same folder and select it
    6. In right pane click [Edit]
    7. In the Parametrize Query section modify the existing query, so that it includes only the desired executables by using the SQL below and change the names
      NOTE: File names should be in lowercase as we use LOWER() to allow work with case-sensitive Databases
      SELECT  r.Guid 
      FROM dbo.[RM_ResourceFile] r
      join Inv_Windows_File wf on wf._ResourceGuid = r.
      Guid
      where LOWER(r.Name) IN ('excel.exe', 'sql.exe', 'helloworld.exe')
    8. Save filter
  2. Open the Hierarchy Managment page
    1. Go to Settings> Notification Server> Hierarchy
    2. Select the Replication tab
    3. Expand the Resources section and search for "File Resources for executable files"
    4. Highlight the rule and click the Pencil icon at the top
    5. In the upper right-hand corner, click the dropdown next to the name and select "Clone"
    6. Name it "File Resources for selected executable files"
    7. Disable the original replication rule "File Resources for executable files" and click "Save changes"
  3. Edit the clone of the original rule "File Resources for selected executable files"
    1. Find newly created rule in the list of rules
    2. Select it and click Edit
    3. Click on resource targets
    4. Remove original target
    5. Create a new target based on the new filter
    6. Add the new target and Save changes

As new executables need to be replicated add them to the SQL in the filter created in step 1