Information and data evidence to collect after a Domino server crash

Products

• Mail Security for Domino

Issue/Introduction

A Domino server where Symantec Mail Security for Domino (SMSDOM) has crashed and the Notes System Diagnostics (NSD) show the faulting process name: nntask.

 

Resolution

NOTE:
Before proceeding, it is very important to locate a file called smspanic.txt in the system.
The file may contain useful information to research in the Symantec Technical Knowledge Base.

 

Gathering information and data evidence for Symantec Technical Support

When Technical Support is contacted for a crash issue, please have the following information and data evidence ready:
 

Required: Description of crash

1. Which process is FATAL? What exception code did the fatal process report?
2. What event, if any, occurs at the same time as the crash?
3. Does the crash occur more than once? Regularly? At a specific time?
4. If at seemingly random intervals, what is the longest and shortest time between crashes?
5. Does load seem to be related?
6. Does LiveUpdate seem to be related?
7. If LiveUpdate is related to the crash, does the crash occur if LiveUpdate is run manually, or when scheduled?
8. Does LiveUpdate retrieve definitions from the public Symantec LiveUpdate servers or a private server internal to the network?
9. Does SMSDOM get definitions in any other way such as replication or via SAVCE or SEP?
10. How many Domino servers have experienced the crash?
11. How many Domino servers are in the environment?
12. Are any non-Domino servers impacted in any way? If so, how?
13. What was the first symptom that led you to suspect the Domino server crashed?
14. Were there any other symptoms related to the crash?
15. Can the crash be reproduced on demand? If so, what are the steps to reproduce? Is there a sample file?

Required: Domino Configuration

1. What is the exact version of SMSDOM or SAVfDOM?
2. What is the exact version, type, and bitness of Domino?
      (Examples: Domino 8.5FP1 Messaging Server [64-bit], Domino 7.0.4 [32-bit])
3. What is the exact version of operating system, including type, bitness, and Service Pack, if any?
      (Examples: Windows2003Enterprise SP2[64-bit], RHEL4AS Update 8 [32-bit])
4. Partitions: Is there more than one DPAR or LPAR?
5. Domino Clusters: Describe each type of Domino cluster where the Domino server is either a hub server or spoke server.
6. Are any of the Symantec databases replicated?
7. Other Clusters: Describe every other type of clustering where the Domino server is either an active or passive node.
8. Realtime Filesystem Utilities: Enumerate each realtime filesystem utility, including name, manufacturer, and exact software version.
      These might be local filesystem antivirus, automatic background disk defragmenters, or realtime mirroring utilities.
      (Examples: Symantec Antivirus Corporate Edition 10.2.1, Diskeeper)
9. Exclusions: Is SMSDOM excluded from the realtime filesystem utilities?
10. Backups: which product backs up Domino server data? Do backups occur within Domino, at a filesystem level, or both?
11. When are backups scheduled?
12. What third party software runs as an extension within the Domino extension manager or as a task within Domino?
13. What recent changes were implemented in the environment before the crash or first crash?

Required: Data evidence for Symantec Technical Support
The following data should be collected from each affected Domino server. 

1. The Domino main configuration file: notes.ini
2. "smspanic.txt" file containing text. The default locations for this file are as follows:
      - Windows: C:\Program Files\Symantec\SMSDOM\smspanic.txt
      - *nix/Solaris: <Domino Data Directory/SAV/savlog/smspanic.txt
3. If the smspanic.txt file is not present, or contains no text, please attempt to reproduce the crash, or wait for the crash to reproduce, with the following forms of tracing enabled:
      - SMSDOM trace logging (Please see the article TECH80022 in the "Related Articles" section below)
      - Semaphore debugging within Domino (Please see this IBM article for more info)
      - Notes.ini set to produce nsd's containing memcheck data
      - Notes.ini should contain the following lines:
          DEBUG_SHOW_TIMEOUT=1
          DEBUG_CAPTURE_TIMEOUT=10
          DEBUG_THREADID=1
          CONSOLE_LOG_ENABLED=1
4. The following NSF files:
      - Log.nsf
      - Sav.nsf
      - Savlog.nsf
      - Savdefs.nsf (if it exists)
      - Statrep.nsf
      NOTE: all NSF files must not have any local access protection set.
5. Compressed archive of the entire folder: <Domino installation path>\data\IBM_TECHNICAL_SUPPORT\
6. Output of running processes/tasks on the Domino server
      - Windows: screen shot(s) of Processes tab within Task Manager with "Show processes from all users" checked, sorted by Image Name
      - *nix/Solaris: ps -ef > processes.txt
7. LiveUpdate configuration
      - Windows: \Documents and Settings\All Users\Application Data\Symantec\LiveUpdate\Settings.LiveUpdate
      - *nix/Solaris: /etc/liveupdate.conf
8. LiveUpdate logs
      - Windows: \Documents and Settings\All Users\Application Data\Symantec\LiveUpdate\Log.LiveUpdate
      - *nix/Solaris: /tmp/livupdt.log
9. Operating System Logs
      - Windows: Application Log and System Log, and Dr.Watson Log, if one exists
      - *nix/Solaris /var/log/messages
10. Operating system tuning and configuration data
       - Windows: a compressed archive containing exports of the following registry keys, should they exist:
          - HKLM\System\CurrentControlSet\Services\Tcpip\Parameters
          - HKLM\System\CurrentControlSet\Control\Session Manager
          - HKLM\Software\Intel
          - HKLM\Software\Symantec
11. Any other data which relates to the crash, such as a Mail.box containing messages which caused the crash

OS-Specific logs required

iSeries - OS/400
Please get the job log of the dead NNTASKprocess, using the following commands:

DSPSPLF FILE(QPJOBLOG) JOB(job_number/Usrid/Job_name)
DSPJOB  JOB(job_number/Usrid/Job_name) OPTION(*SPLF)

for more information please refer to: http://publib.boulder.ibm.com/infocenter/iadthelp/v6r0/index.jsp?topic=/com.ibm.etools.iseries.rse.doc/tasks/tdspjoblog.htm


When you contact Symantec Technical Support, provide as much specific information as possible.
Your support agent can work with you to transfer your log information to them.



Additional References
The following IBM documents can help understand NSD files:

• Title: NSD and Memcheck for Windows - Selected Versions - Document 4003878
  URL: http://www-01.ibm.com/support/docview.wss?uid=swg24003878
   
• Title: Overview of NSD Analysis on UNIX and S/390 - Document 7003396
  URL: http://www-01.ibm.com/support/docview.wss?uid=swg27003396
   
• Title: Memcheck: What Is It and How Does it Work? - Document 1086330
  URL: http://www-01.ibm.com/support/docview.wss?uid=swg21086330
   
• Title: Using NSD - A Practical Guide
  URL: http://www-10.lotus.com/ldd/dpiblog.nsf/dx/Lotusphere%202008-HND107-Handout.pdf/$file/Lotusphere%202008-HND107-Handout.pdf

Utilities to crash client and server
http://www-10.lotus.com/ldd/sandbox.nsf/0/ccca569e9b9810d185256c5300448031




 

Applies To

NNTASK is SMSDOM's main process name.