Out of memory errors are often not caused by a lack of memory. Often they are caused by a configuration situation and this puts stress on the Notification server. There are a few things to look for.
1) Altiris Agent.
Change your Altiris Agent check in time. Set it to 4 hours and a 1 week basic inventory. Basic inventory gathers static data that doesn't change often. Such as Windows operating system, Mac Address or IP address. Having this updated every hour creates unnecessary traffic and processing for the Notification Server. This setting is suggested especially if you have more than 500 clients checking into the NS directly. If you have Site Servers, or Task Servers I would recommend that you put those check-ins time at 2 hours in length.
Make sure that you have a maintenance plan setup on your SQL DB and that you also run Disk Defragmentation every 1-2 weeks. SQL Access and time outs errors have occurred because their SQL DB files are in 45,000 pieces. It's very Important to index your SQL DB. That being said if your DB is in tens of thousands of pieces even after you run the Maintenance plan it's going to aid you very little. Maintenance puts your data in the correct order for the database, it does not move the data itself on the drive to be in a contiguous format.
3) Event Processing
Check your event folders, they are located at <Install Dir>Program files\Altiris\ Notification Server\NSCap\EvtQFast (EvtQSlow, EvtQLarge)
If you have more than about 15,000 files in here. (total starting from NSCap and it's sub directories) IIS doesn't process things very well. When you hit 20,000 it does one of two things. 1) it stops processing all together. or 2) it tries to do too much and causes an overload. You'll see your W3WP process at nearly 70% or more of your CPU usage.
To circumvent this delete the contents of the bad folders in the event folders.
4) Bug fix
There is a known issue with IIS 6. If the Footprint for the w3wp Service exceeds a certain amount of undocumented memory usage it begins to grow exponentially until it locks up. Microsoft has address this issue in an hotfix for IIS 6 http://support.microsoft.com/kb/2418042
5) ASP.NET errors in the event logs
This section is for Notification Server 6 Platform only!
C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322> And run 'aspnet_regiis -lk'
This should show something similar to the following
If it does not type in the following command
This should help alleviate any issues you may have, if it does not there may be a bigger issue with the .Net Framework and it may require to have 1.1 to be re-installed
If you do a repair of .Net 1.1 you'll need to make sure that In IIS In the "Web Service Extension" Folder that ASP.NET 1.1 Is allowed.
If you follow the above Guideline you should find that it should address most performance problems, if it does not you should consider that it might be related to how your environment is setup.