You observe that Symantec Mail Security for Domino (hereafter SMSDOM) 8.x crashes when (a) it runs as a 32-bit application on a AIX platform working with the Domino server, and (b) nntask runs out of the process heap memory.

If you execute show stat platform.process.nntask.* command on the Domino console, you'll see Platform.process.nntask.1.HeapFreeKBytes is almost 0 or significantly low.

The following is the NSD log example of the process memory heap information observed on such a crash:

** Proc nntask/13566116 : heap addr = 0x20000000, size=805302704
// snip...

** Proc nntask/13566116 : heap addr = 0x20000000, size=805302704
// snip...

<@@ ------ Notes Memory -> Usage Summary -> Process Heap Memory Stats :: [  nntask:13566116] (Time 04:00:15) ------ @@>

    TYPE        : Count    SIZE      ALLOC     FREE     FRAG OVERHEAD  %used  %free
    Static-DPOOL:     4   2097152    377756   1717712      0     2320    18%    81%
    VPOOL       :     1     64512       500     62388      0     1632     0%    96%
    POOL        :     3    110240     46858     57784      0     5622    42%    52%
    Overall     :     4   2097152    257584   1837884      0     9574    12%    87%

Process Heap: size=805302704, limit=2147483647, %used = 37
Process Heap: size=805302704, %used (by Notes) = 0

nntask has run out of the process heap memory, which is by default 512MB for a 32-bit environment.

Increase the AIX's heap segment size available for nntask as follows:

IMPORTANT: Stop the Domino service and the Mail Security for Domino first, and then follow BOTH of the step (1) and the step (2).

(1) Tune the shared memory for Domino

Edit notes.ini and set the parameters as shown below:

NSF_BUFFER_POOL_SIZE_MB=400
NSF_DbCache_Maxentries=1500
NLCACHE_SIZE=67108864
AIX_LIMIT_SHM_SEGMENTS=8
Platform_Statistics_Disabled=0
SAVBufferSize=0

(2) Increase the process heap's segment size from 2 (default) to 3

Increase the AIX's process heap memory from 512MB (256MB * 2, default) to 768MB (256MB * 3).

1. Log in to AIX as the root user.
2. On the terminal, confirm the heap's maxDATA information with the following command.
   
   dump -X 32 -ov /opt/Symantec/SMSDOM/bin/nntask
   
   You'll see something similar to the following example.
   
   //snip...
   maxSTACK maxDATA SNbss magic modtype
   0x00000000 0x20000000 0x0003 0x010b 1L
   
3. Set the process heap's maxDATA as follows.
   
   /usr/ccs/bin/ldedit -b maxdata:0x30000000 /opt/Symantec/SMSDOM/bin/nntask
   
4. Check the maxDATA for the updated value.
   
   dump -X 32 -ov /opt/Symantec/SMSDOM/bin/nntask
   
   You should see that maxDATA is now 0x30000000.
   
   // snip...
   maxSTACK maxDATA SNbss magic modtype
   0x00000000 0x30000000 0x0003 0x010b 1L
   
5. Login as the Domino User and run the export command:
   
   export LDR_CNTRL=MAXDATA=0x30000000
   
6. Start the Domino service and the Mail Security for Domino.
    

See to the following documents for the Domino parameters.

Title: NSF_BUFFER_POOL_SIZE_MB parameter: Best Practice (Japanese)
http://www-01.ibm.com/support/docview.wss?uid=swg21463566

* For the AIX the recommended value is 512MB; however, 400MB is the IBM-recommended value based on the related case.

Title: NSF_DbCache_Maxentries  (Japanese)
http://www.bcom.co.jp/help/help7_admin.nsf/2e73cbb2141acefa85256b8700688cea/50c1debdc7fbf8a94925707a0042e1fa!OpenDocument

Title: NLCACHE_SIZE (Japanese)
Name Lookup Cache (NLCache_Size) default value
http://www-01.ibm.com/support/docview.wss?uid=swg21462953

Title: (FYI) AIX_LIMIT_SHM_SEGMENTS parameter and its shared memory management for Lotus Domino for AIX (Japanese)
http://www-01.ibm.com/support/docview.wss?uid=swg21464342

Title: Platform_Statistics_Disabled (Japanese)
http://www.bcom.co.jp/help/help7_admin.nsf/2e73cbb2141acefa85256b8700688cea/07d4c2d979cee1424925707a0042e265!OpenDocument