Out of memory errors are often not caused by a lack of memory. Often they are caused by a configuration situation and this puts stress on the Notification server. There are a few things to look for.
1) Altiris Agent.
Change your Altiris Agent check in time. Set it to 4 hours and a 1 week basic inventory. Basic inventory gathers static data that doesn't change often. Such as Windows operating system, Mac Address, and IP address. Having this updated every hour creates unnecessary traffic and processing for the Notification Server. This setting is suggested especially if you have more than 500 clients checking into the NS directly. The exception to this is if you are monitoring user/owner information for who spends the most time logged into resources.
If you have Site Servers, or Task Servers It is recommended that you put those check-in times at 2 hours in length. It is advised that your Site or Task Server is at half of the time that your agents check in at.
Make sure that you have a maintenance plan setup on your SQL DB. It is also advised that you run Disk Defragmentation of the drive that houses your SQL Databases every 1-2 weeks. SQL Access and time out errors have occurred because SQL DB files are in thousands of pieces. It's very Important to index your SQL DB. Maintenance puts your data in the correct order for the database, it does not move the data itself on the drive to be in a contiguous format.
3) Event Processing
Check event folders, they are located at <Install Dir>Program files\Altiris\ Notification Server\NSCap\EvtQFast (EvtQSlow, EvtQLarge)
If you have more than 15,000 files in here. (total starting from NSCap and it's sub directories) IIS doesn't process things very well. When you hit 20,000 it does one of two things. 1) it stops processing all together. or 2) it tries to do too much and causes an overload. You'll see your W3WP process at nearly 70% or more of your CPU usage.
To circumvent this delete the contents of the bad folders in the event folders.
4) Bug fix
There is a known issue with IIS 6. If the Footprint for the W3WP Service exceeds a certain amount of undocumented memory usage it begins to grow exponentially until it locks up. Microsoft has addressed this issue in an hotfix for IIS 6 http://support.microsoft.com/kb/2418042
5) ASP.NET errors in the event logs
This section is for Notification Server 6 Platform only!
C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322> And run 'aspnet_regiis -lk'
This should show something similar to the following
If it does not type in the following command
This should help alleviate any issues you may have, if it does not there may be a bigger issue with the .Net Framework and it may require to have 1.1 to be re-installed
If you do a repair of .Net 1.1 you'll need to make sure that in IIS in the "Web Service Extension" Folder that ASP.NET 1.1 is allowed.
Following the above Guidelines should address most performance issues.