Creating a hierarchy adds a tremendous amount of complexity to the process. There are also a lot of misconceptions regarding what hierarchy is and what it can do. A hierarchy allows for scaling with a central management structure from the top down.
The decision to move to hierarchy should be based upon the number of nodes reporting to a single NS and whether performance is acceptable. Maintaining a single NS, if possible, is a better solution than moving to hierarchy due to the complexity and increased administrative overhead.
For other articles regarding Hierarchy/Replication, see "Hierarchy and Replication"
What are the most frequently asked questions about Hierarchy.
1. Is Hierarchy designed to address performance, scalability or availability limitations with either the Notification Server or associated solutions?
No. Hierarchy is primarily designed to reduce the total cost of ownership (TCO) of managing Altiris software and solutions across multiple Notification Servers. It is not designed to address performance, scalability or availability limitations with either the Notification Server or associated solutions.
However, Hierarchy can be used to link multiple Notification Servers together to act as a single unit in terms of configuration and data management. This configuration enables customers to scale-out their enterprise, by adding additional Notification Servers, to distribute traffic and load as the number of managed resources increases.
2. Why does Hierarchy use a hierarchical style network topology as opposed to a fully connected topology?
Hierarchy uses a hierarchical style network topology to maximize scalability and minimize conflict management.
For a fully connected network, each server is required to replicate to each other server within the same structure. Each time a server is added to the structure, this is one more server that each other server must replicate too. Eventually the cost of replicating to every other server will start to negatively impact each server in terms of load and network bandwidth.
For a hierarchical network, each server is required to only replicate to its immediate parent and children. If another server is added to the structure, then only the servers that are an immediate parent or child to the new server are impacted. In this manner, hierarchical networks can scale out and support significantly more nodes than fully connected networks.
Another benefit of hierarchal networks over fully connected networks is the simplicity of conflict management. In a fully connected network, where replication is ad hoc, it is extremely difficult and expensive to synchronize changes made to shared data made on different servers. In a hierarchical network, with directional replication (i.e. up or down), synchronization is both trivial and predicable
3. Does Hierarchy always replicate objects and data even if they haven't changed since the last replication?
No. Hierarchy supports both Differential and Complete replication.
Differential replication only replicates objects and data that have changed since last replication. This process involves generating and comparing cryptographic checksums of objects and data at each replication endpoint. Replication is necessary only if either the replication data is absent from the destination server or if the cryptographic checksums do not match.
Complete replication replicates all objects and data regardless of whether or not they have changed since last replication.
4. How does Hierarchy impact licensing?
Specific licensing policy implementations such as Agent Based and Inventory Based licensing must factor in whether an NS is participating within a Hierarchy when calculating in-use licenses.
5. Should events be replicated within a Hierarchy?
Due to the large number of events raised on each Notification Server, event replication should be avoided and only used under very specific conditions.
As a general guideline, configuration and management items, security settings, resources (in terms of inventory data) and packages change infrequently. So once they are initially replicated, the cost of synchronization (in terms of bandwidth and load) is dramatically reduced, as only objects and/or data that have changed since last replication needs to be resent. However, event data is different in that it is volatile and continues to accumulate over time. As well, event data is typically not purged until either size or time restrictions are exceeded, thus event tables tend to contain very large amounts of data. So the dynamic nature of events, combined with their sheer volume, makes replication and subsequent synchronization very expensive in terms of bandwidth and server load.
6. How does Hierarchy handle duplicate resources?
As a hierarchy consists of multiple individual NS machines which may perform actions like Active Directory imports or Domain Resource Discovery, the same resource can be independently created at multiple locations. As these resources are created independently, they have different unique identifiers. The hierarchy infrastructure has specific handling to deal with this scenario, but it can take a while for everything to stabilize on a single unique identifier.
As an example scenario, we have 4 hierarchy NS machines, A, B, C and D. B and C are an immediate child of A, and D is a child of C. Discovery is run at both B and D and so a specific machine we will call ‘ManagedClient’ is known to both of these NSs.
The agent is then rolled out to ‘ManagedClient’ from NS B, so NS B is managing this machine.
The first night, the replication schedules will run (assuming default schedules are set up). B will replicate ‘ManagedClient’ to A. D will replicate ‘ManagedClient’ to C.
The second night, the replication schedules will run again. This time, C will replicate ‘ManagedClient’ to A. Following the data flow you can see that A already has ‘ManagedClient’ with the unique identifier from the discovery at B, but the resource being replicated from C will have a different unique identifier. However, during the replication from C to A, the existing resource is detected because they have the same name.domain ResourceKey. (name.domain is one of the registered resource keys which support automatic detection and merge.) The existing resource will be merged in to the copy replicated from C.
When the merge occurs in the hierarchy case, special code is run to register the details of the merge, the resource key names and values which caused the match and the unique identifier of the resource which was being replicated at the time. This unique identifier is the 'winner' of the merge and hierarchy will now attempt to change all instances of the resource to use this unique identifier.
Every 15 minutes (quarter hour schedule) the list of registered merge events are loaded up and sent to all machines in the direction of the source of the resource which caused the event. In the specific example case, this means that within 15 minutes after hierarchy replication has imported ‘ManagedClient’ from C, an event will be raised down the hierarchy containing the details. Specifically it will be sent to machines B, C and D, as they are all children of A (directly or indirectly).
When this event is received at C shortly afterward, it checks for any resources which match on the resource key. It finds ‘ManagedClient’, but its unique identifier is the same as the unique identifier which caused the merge at A. Therefore nothing happens. The same happens when the event is received at D.
When the event is received at B, it checks for any resources which match on the resource key. It finds ‘ManagedClient’, but this time the unique identifier is not the same as the one which caused the merge at A. This causes two things to happen. Firstly the resource has its unique identifier changed to the one which caused the merge. Secondly the old unique identifier is added to an internal blacklist, because old ‘ManagedClient’ resource at B indicated that it was managed locally.
Once the old unique identifier is added to the blacklist, any inventory sent to NS B regarding ‘ManagedClient’ will result in an error being returned if the old unique identifier is reported.
When the agent next requests client config reporting the old unique identifier, it will be found in the internal blacklist and a specific error code is returned to the agent which indicates that it needs to renegotiate its unique identifier with the NS.
The agent will call create resource, passing up its resource keys, which will match against the ‘ManagedClient’ which now has the unique identifier which was originally discovered at D. This unique identifier will be returned to the agent, and it will subsequently request client config/send basic inventory using this new unique identifier.
And finally, everywhere is referring to ‘ManagedClient’ by a single unique identifier.
7. How does Hierarchy handle moving a managed resource between Notification Servers within the hierarchy?
Hierarchy has specific handling so that an agent which reports to a given NS, is considered local to that NS and managed by that NS, even if that agent is known elsewhere in the hierarchy by discovery.
A special case of interest occurs when an agent is moved to report to a different NS, after it has been reporting to an original NS for a while. Since the original NS is unaware that the agent has stopped reporting, it continues to believe that the agent is locally managed. Hierarchy has to detect this scenario and fix it up.
In example we will take a 4 node hierarchy, A, B, C, D. B and C are immediate children of A and D is a child of C.
Initially, ‘ManagedClient’ has been rolled out at NS C. ‘ManagedClient’ is considered local at NS C, even if discovery has been run at NS D. It is also considered to be a managed resource at NS C, even if NS D thinks it is an unmanaged resource.
As this scenario has been in operation for several days, ‘ManagedClient’ is also known at NS A. ‘ManagedClient’ is considered a managed resource at NS A, because one of its children is managing the resource. It is not considered local, because the resource has been replicated from NS C.
Now the owner of ‘ManagedClient’ is transferring to a different office which is managed by NS B. The agent is reconfigured to point to NS B, and it sends basic inventory. This causes NS B to consider ‘ManagedClient’ to be managed and local.
(If NS B does not already know about this resource, see the above FAQ entry about how duplicate resources are handled, that process will occur first at this point. After that a similar but different set of events will occur, compared to the following description.)
On the next hierarchy replication run NS B will send the details about ‘ManagedClient’ to NS A, including the fact that it considers the resource to be locally managed. On receiving these details, NS A will notice that it knows this resource as being managed from NS C. It will consider this scenario to be a conflict. In response to a conflict it first clears its record of whether the resource is managed, and registers an event to notify the children that a conflict has occurred.
Every 15 minutes (quarter hour schedule) the list of registered conflict events are loaded up and sent to all machines in the direction of the source of the resource which caused the event. In the specific example case, this means that within 15 minutes after hierarchy replication has imported ‘ManagedClient’ from NS B, an event will be raised down the hierarchy stating that a conflict has occurred for that resource. Specifically it will be sent to machines B, C and D, as they are all children of A (directly or indirectly).
When the event is received at B and C shortly afterward, they clear their managed flag for the resource. At D the resource is not marked managed, so no change occurs. Note that at this point we cannot specifically know which of B or C is managing the resource now, so we have to clear both.
On the following hierarchy replication from NS D to NS C, it will pass up the details of an unmanaged resource and NS C will accept NS D as being its source of information about ‘ManagedClient’. It will now neither be local or managed according to NS C.
When ‘ManagedClient’ next reports basic inventory, NS B will mark the resource as locally managed once more.
On the following hierarchy replication from NS B to NS A, it will pass up the indicators that the resource is managed locally and NS A will record the resource as managed remotely at NS B.
Finally, when NS C next reports to NS A, NS A will ignore source and managed status details as it knows the resource is locally managed at NS B and NS C is not claiming otherwise.
8. How does Hierarchy address server over-load issues during a replication cycle?
Hierarchy has a built-in throttling mechanism which kicks in to ensure that data will be replicated even if a server is overly busy at any given point of time. This will result in replication taking a longer time to complete. You will see replication status messages in the NS Log Viewer.
9. When replicating packages via hierarchy or stand-alone replication will it replicate the package files and snapshot.xml files as well?
No, the snapshot and package files stay at the source NS, only the item and data representing the resource is sent. The data representing the resource is what would be obtained if exporting the package to XML.
10. How does a Package Server download a replicated package?
The Package Server (PS) sends a GetPackageInfo request to its local NS. The local NS sends the Package Server a redirect URL to the GetPackageinfo.aspx page on the Parent NS. The Package Server directly contacts this page on the parent NS, and is returned ready Package Server sources which are in the same site as the parent NS. If no Package Servers in the same site as the NS are ready with the package at the parent NS then no sources are returned to requesting PS.
Note: The child package server will not be returned ready package server sources which are not within the parent NS site.
Additionally the parent NS will return to the requesting child NS its configured Agent Connectivity Credential and password.
This ensures that if the child PS and Parent PS are in different non-trusted domains, that the child PS will use the appropriate credentials when connecting to download the package. The Agent Connectivity Credential passed down to the child PS, will be stored per package inside the Package.xml file.
11. For the following scenario; in a three tier hierarchy, a Package is created at NS A, replicated to NS B, replicated from NS B to NS C. If a Package Server (PS) from NS C needs the package, will it ever go to PS’s connected to NS A?
No, a PS will only ever go to its immediate parent to get a package.
It is assumed that at least one PS on NS B will have the package at some point, and if not the PS on NS C will backoff until that time.
12. Should a user from a bottom level NS be able to view the entire hierarchy structure via the Hierarchy Management page?
Any given node in the hierarchy has the ability to see its immediate parent and its entire sub-tree. A node should never be able to see beyond its parent since this is out of their "responsibility" area. Because a parent node determines what goes down the hierarchy, they can see everyone below them.
13. Do packages have Resource Keys?
No, and they will not.
14. When I configure a hierarchy/replication rule to run on a custom schedule, what type of replication occurs, a complete or differential replication?
A differential replication is triggered.
15. If a computer and all its inventory class are replicated, and then later one inventory class for the computer is modified/changes, and a differential replication is carried out, does this mean all inventory classes for the resource are sent again, or only the specific inventory class that changed?
Only the specific inventory class is re-sent.
Note: If the inventory class is multi-rowed (such as the data class AeX_AC_Client_Agent) then all rows are re-sent.
16. If an item or resource has been replicated many levels up or down the hierarchy, how do I investigate which NS it originated from?
As soon as a NS is added to a hierarchy structure, all items will have a new hierarchy tab within their item properties.
The tab lists ‘Original Source’, which is the server name where the item originated from (this could be many levels up the hierarchy).
If the item originated locally, then Original Source will be displayed as the local NS name.
Also displayed is ‘Replication Source’ which is the actual parent server which sent the item to the current server. This will be left blank if the item originated locally.
17. If I create a SWD task and package, why is it that when I specify to replicate All Configuration and Management Items, it sends both the SWD task, and the SWD Package, even though I have not specified for packages to be replicated in the resource replication rules section?
This is because the SWD Package is considered a ‘dependent item’. The SWD task requires the associated SWD Package, otherwise it will be unusable at the destination NS.
18. If I replicate a resource (Computer, SWD Package, User, Site or Subnet) and then I delete the resource at the source and replicate again, should it also delete the resource at the destination?
It will only do this for replicated Configuration and Management items, not for any resources or events. (See HOWTO42295
regarding Hierarchy Singular Task for more information)
19. Should core objects be replicated?
Core objects are replicated if their details are changed and they aren’t marked non-replicable. This allows for settings information to be replicated across the hierarchy.
20. When a site is replicated down the hierarchy, will it also replicate any subnets that are contained within that site?
No, it does not replicate the subnet resource; it only replicates the resource association between the site and the subnet. Therefore if the subnet already exists at the lower node, then it will be automatically added to the replicated site.
21. If a resource is replicated which has custom data classes associated with it, that do not exist at the parent, will it create those data classes at the parent when the resource is replicated?
Yes the dataclass is sent first and created at the destination, then the data is sent afterwards.
22. How do I know whether an item should be considered for replication?
Each Item has ‘Item Attributes’. These can be seen by right clicking on the item and going to Properties. If the attributes of an item are either ‘System’ or ‘No Replication’ then the Item should never be considered for replication. However, the security placed on these items will still be replicated, regardless.
23. When ‘Replicate Now…’ is selected inside the context menu for an item or folder at a parent node, does this trigger a Complete or Differential replication?
A Complete replication is triggered. The Item(s) or folder(s) will always be resent when a ‘Replicate Now…’ action is triggered, regardless of whether anything has changed since last replication.
24. Does Hierarchy/Replication send Resource History data?
There is no Resource History replication. However in future versions (8.1x) there will be a setting added to Coresettings.config. The setting "EnableDataClassHistoryForReplicatedData" can be turned on via ".\Notification Server\Bin\Tools\NSConfigurator.exe". This setting will cause changes to data classes that have Resource History retention turned on to be stored for replicated computers.
25. What account credentials can be specified for the ‘connect to’ and ‘connect back’ credentials when setting up a hierarchy, or creating a stand-alone replication job.
The account specified must be a member of the Symantec administrators Security Role on the NS you are trying to authenticate with. Being a local administrator of the NS machine is not sufficient.
26. Do Emergency Policy Update jobs interrupt any current replication jobs or do they wait for currently running jobs to complete before executing?
Each EPU triggers an urgent hierarchy job, like a replicate now type operation. These jobs once triggered act independently of each other, and are not queued. So they would not have to wait for any other jobs to finish. However overall there is a still a maximum number of simultaneous jobs, which is controlled by the core setting ‘ReplicationMaxJobThreadCount’ - set to 3 by default.
A new replication job will never be triggered when already above that limit. The job will be put in a queue until it is under the limit and then triggered. The urgent hierarchy jobs are selected as priority to be executed above any other regular jobs that may already be in the queue.
27. What the numbers under the ‘direction’ column for the ‘ItemReplication’ table means?
The values in this column mean the following:
1 – Object replicated up
2 – Object replicated down
3 – Object replication both up and down
0 – Undefined
28. What the values under the ‘ImportMethod’ column for the ‘ItemImportMethod’ table means?
The table called ItemImportMethod lists each Item and the method of which it came to exist in the database.
The ‘ImportMethod’ column can have the following values:
0 – Standard local object
1- External Object, e.g. item imported from xml.
2 – Originated from standard Replication
3 – Hierarchy managed item
When a hierarchy managed item is reset due to the node being removed from the hierarchy structure, the ‘ImportMethod’ column for the item is changed from 3 to 0.
29. Can you enable Hierarchy at any time?
Basically if you already have a system in place can you break out into Hierarchy and start adding Child Ns’s?For a hierarchy to be created, each node in the hierarchy must have an off box site server with package services enabled in the same site as a the SMP. If this is configured, then you can add child and/or parent nodes.
30. Would it be better to split off two separate Hierarchies and of a master that isn't used except for Global policies? Is it possible to make our current NS server a Child NS and would that be best practice? This way we would have one Clean master with Two child NS’s.
Normally, you want client facing SMPs to be your child SMPs with the parent SMP being the source for management, reporting and Asset. If you currently have an SMP that is child facing, I would definitely make it a child to another non-client facing SMP.
Clients will need to be configured between the child SMPs. Whether you decide to organize these based upon organizational groups or some other way is up to you.
31. Can you use the same SQL instance on the Database server for multiple NS’s or child NS’s?
With 32-bit SQL we required that each database be in its own instance. Even with the performance benefit of 64 bit I would still recommend creating different instances for each database. The SMP utilizes the TEMP database heavily and if you have all databases using the same instances then all of the databases will be competing for use of the TEMP database.
If your SQL box is beefy and running 64-bit, I would use the same server but make sure each database is configured in its own instance. Each instance will therefore have its own TEMP db.
32. What global policies can you push from the master NS server? For instance can you set patch policies globally from the master NS and replicate down.
Items replicated down and resources replicate up by default. Policies are considered at item and they replicate down along with any dependencies. So yes, you can create the policies on the parent and they will replicate down to each of the child SMPs.
As an FYI, replicated items are by flagged as read-only. Once an item has replicated, changes need to be made at the parent.
Patch policies do replicate down. Their implementation is rather complex in my opinion. The child SMPS must replicate their languages up. The parent replicates the policy configurations down based upon the languages. PMImport is triggered on each child SMP. There is a lot of synchronization that has to occur but it seems to work well.
33. Do Software packages replicate down to the Child NS’s?
34. How long does Replication take on average?
The XML for the packages replicate down to the child SMPs. The actual files do NOT. Ownership of these replicated packages is maintained by the parent. Package Servers on the child make a packageinfo request to their NS and are redirected to the parent. The parent will then hand off codebases to its offsite package server where the package should be ready. The offsite package server on the child then downloads the files from the offsite package server on the parent.
That varies on the size of the database, the amount of the data and the hardware capabilities. A complete replication can take hours (4-8). A differential may take a couple of hours.
We recommend NOT using “complete” replication because of the burden it is on the network and servers. A differential will only replicate those items that are necessary and does so in a much more efficient way.
35. What can’t you replicate down to the Child NS?
The general rule is that resources replicate up and items replicate down. Each solution determines its replication process. Some solutions are not replicable. Asset is one of them. Items (policies, packages, filters, targets, etc.) will replicate down. Computers, Subnets, Files, replicate up.
You can configure replication rules to replicate resources down but to have resources replicate in both directions is unsupported.
There are a couple of other scenarios where resources are “relocated up” to the parent and then replicated back down. An example is subnet replication. Subnets are created by basic inventory when agents check into their client facing SMPs via Inv_Aex_AC_TCP_IP data. This data is “relocated up” meaning that each subnet resource will only replicate up once and once it does, the ownership for the subnet is changed to the parent. Subnet assignments to sites are then made on the parent. Subnet and Site data are then replicated back down to all child SMPs.
Organizations views and groups, security roles, permissions and privileges also replicate down.
If AD import is used at the parent, the organizational structure replicates down but the scope membership does not because it is based upon resource data (resources should only replicate up). Connector rules must be used to import the dataclass data for the computers down to the child SMPs without also replicating the resources down.
Organizational views and groups created manually on the parent replicates down as well as its scope membership since scope membership for this process is maintain in the scopemembership table. However, if organization structure is managed from the parent, it must always be managed from the parent because changes made directly to the child SMPs will be overwritten by what exists on the parent when replication occurs.
36. Do all the servers need to be at the same patch level as far as CMS goes?
37. What are the top caveats or Gotcha’s Hierarchy?
Yes. The process for upgrading servers is to disable replication on all servers, upgrade the child SMPs and then the upgrade the parent. Once the upgrade has completed, reenable replication.
Hierarchy is not a synchronization process between servers. We do not track the deletion of resources between the child and the parent but only with certain resource types. This list will increase with 7.5.
Security has its own way of replicating.
Security is NOT included in the hash calculation of an item so there isn’t any way to know whether the security of an item on the parent matches that of the child. If the permission replication process fails, there isn’t any way for the NS to verify and report on this. This has much improved in 7.1 however since roles are now considered items and permissions are included in the XML of an item. So if the item changes, its permissions are also replicated.
Scopemembership is confusing. There is a process that must be followed if creating the organization structure manually or via AD import. There is a KB for this.
Application metering reporting currently doesn’t work correctly. A process must be followed to replicate certain dataclass data up to the parent in order for metering reporting to work. There is a KB for this.
Replication is administratively intense. It will require a basic knowledge of how replication works. Replication jobs can and do fail for numerous reasons which may require manual intervention to clean up these hung jobs and orphaned data.
Tasks execution from the parent down to targets on the child SMPs are unidirectional. Tasks will replicate down to the child SMPs but schedules created on the parent execute on the parent. The parent will send down replication jobs down to the child SMPs for those clients that are in the task’s target. The tasks will run on the agents in the target but there isn’t any reporting back up to the parent to indicate if the task ran successfully. Nothing will be reported back to the child SMP either. Tasks can be created on the parent but should be scheduled and executed on the child SMPs for accurate reporting.
38. Can you report on everything from the Master NS?
No. Not everything. The example above with tasks is an example where the task execution is not reported back up. Built in reporting is not very robust when it comes to reporting across the hierarchy. The reason is that in order to report on data across all servers, either all of the data for all servers must exists in one location OR linkedservers must be used to allow for the pulling of data from each database. Since we don't know what each customer’s hierarchy looks like it’s impossible to have canned reports to accomplish this. IT Analytics may help bridge this as it creates linked servers and pulls the data into a central location.
There is also a KB for creating reporting for package servers across all SMPs. This process could be used as an example for creating other reports. The KB is HOWTO83775.
The Patch data isn’t replicated up. Instead the reports on the parent open up on each of the child SMPS.
With 7.x you can replicate event data up though we don’t recommend it.
39. Can you replicate Tasks or image jobs?
As mentioned above, you can replicate tasks, however; schedules should be configured and ran on the child SMP for accurate reporting. Policies and their schedules replicate down and run on the child SMPs.
Each solution is responsible for their own replication implementation. I do know that the DS team recommends NOT using hierarchy for pushing images. The recommendation is to configure and use imaging specifically from the child SMPs
40. Does the deploy anywhere driver database replicate out?
I believe so.
41. Why does it take up to 3 days for some items to replicate down the Hierarchy?
Some items have many pieces that need to be replicated, if any portion is not fully ready when the first replication runs it will not be fully functional on the child servers until the next replication schedule has completed.
One of the most common items are packages, if they have not downloaded to the site server in the same site as the SMP server and returned the codebases when the first replication runs this situation will occur.
42. Why does the Hierarchy topology page show High alert status for recent replications?
The data used to show the latest alert status includes failures that occurred for items that are not replicable and items that did not exist for valid reasons. This can skew the count in a large way. Run other reports to verify replication status rather than relying on the one on the Hierarchy topology page.
43. Why do my Hierarchy jobs show multiple runs and can I clear them up?
Follow the KB for replication cleanup. KB: TECH141847
44. What is the default timeout for a replication job?
45. How many replication jobs can run at any given time on a server?
46. Is it recommended to run a complete replication periodically?
Only schedule a complete replication job as needed, such as when a network failure has occurred during a differential, or some other replication failure occurs, or as guided by support. The differential replication is much more efficient without the performance hit to the servers.
47. Can I set a Standalone Replication rule to do a complete replication?
No. Standalone Replication rules always run in differential mode.
48. When I right-click a folder and select Hierarchy/Replicate Now, will the replication run as Complete or Differential?
49. When I right-click a Node in the Hierarchy Topology and select Replicate To, will the replication run as Complete or Differential?
50. What are the concerns about setting the Configuration and Management Items to "Custom"?
We do not recommend that the Configuration and Management Items "Custom" setting be used due to hidden items that can't be selected and the volume of items that would need to be selected to make a custom process work. Customers may find that items they expect to replicate don't because of the "custom" setting.
The recommended setting is "All". Selecting "All" will replicate only those items that need to be replicated during a differential.