A complete guide to logging and data gathering for Symantec Monitor Solution.
Article ID: 181615
Updated On:
Products
Issue/Introduction
A complete guide to logging and data gathering for Symantec Monitor Solution.
Resolution
Prelude:
For brevity purposes many acronyms are used in this document. The first occurrence of any terminology used in this document will include the complete spelled name of the term accompanied by the acronym in parenthesis. For easy reference below is a list of terminology used in this document and the associated acronyms:
SMA: Symantec Management Agent.
NS: Notification Server.
SMP: Symantec Management Platform.
SMP Server: Notification Server on which the Symantec Management Platform is installed.
Introduction:
This guide covers how to collect all of the available forms of logging for Symantec Monitor Solution, what they are used for, and examples of when they should be collected for troubleshooting purposes. This document covers the following forms of logging:
-Monitor Agent Logs
-PPA Logs
-Agent Logs
-Server Logs
-Verbose Agent and Server Logs
-Policy Data: exported from servers and client's policies.
-NT Event Logs
-WMI Diagnostic Data, Testing, and Logging
-Process Monitor Logs
-SQL Server Error Logs and Trace
Monitor Logs - How to collect:
Please follow the steps below to collect the Monitor Agent Logs:
1.) Please visit http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx and download
dbgview.exe
2.) Click Start > Run > regedit.
3.) Once regedit is running navigate to the following key
HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\eXpress\Monitor\Debug.
4.) Record the original settings in this key so that they may be changed back after logging is
complete.
5.) Set the following keys as shown:
"Error"=dword:00000001
"Information"=dword:00000001
"Profile"=dword:00000001
"Trace"=dword:00000001
"Warning"=dword:00000001
6.) Navigate to the download location of the dbgview.exe program and double click it to run it.
7.) Once dbgview.exe is running click Capture > Capture Global Win32.
8.) Once Capture Global Win32 has been enabled there will be many more events that appear in
the dbgview log area. Once this is running perform the task in question so that the
events associated with the even in question will be logged.
9.) Once the task has been completed while monitor logging has been enabled click File > Save.
10.) Give the file an name and save it in the default .LOG format.
11.) Navigate back to HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\eXpress\Monitor\Debug and
restore the default settings that were recorded earlier.
PPA Logs - Introduction
PPA stands for pluggable protocol architecture and is used for storing information that is required to communicate with computers and other devices using standard network
monitoring protocols. The Pluggable Protocols Architecture unifies the configuration of protocols across the Symantec Management Platform. To capture PPA logs the registry
must be edited.
PPA Logs - When to collect:
There can be a number of reasons to collect PPA logs. General reasons include: PPA error referenced in other logs, PPA error in Symantec management console, issue being caused by unknown source
and PPA is a good place to look for issues.
PPA Logs - How to collect:
1.) Open Windows Registry Editor via: Start > Run > type: regedit and click OK.
2.) Access the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\PluggableProtocols
3.) Right-click PluggableProtocols and create a new Key named "Debug".
4.) Right-click Debug and add a DWORD value named "Level" with a value of 0. This value alone will allow the debug output to be viewed by a 3rd party debugging tool
(e.g. Microsoft's DebugView, available with the SysInternals suite). If the debug output needs to be written to a file, add these additional values under the Debug key:
"Dest"=string: with a value of file([absolute path and file name]) e.g. file(C:\temp\ppa.log)
"Format"=string: %time%($MM/$dd/$yy $hh:$mm:$ss) [%thread_id%]
5.) If on the Symantec Management Platform (SMP) server, restart the Altiris Object Host Service. If on a Site Server, restart the Symantec Management Agent service
- Note: For additional AMT logging, include the following in step 4 above:
Right-click PluggableProtocols and choose to create a new Key named "AMT".
Right-click AMT and choose to add a DWORD value named "TraceLevel" with a value of 31.
Note: PPA debug provides intensive trace output data. Debug Viewer, or logs, will grow large rapidly. It is recommended to disable debug output immediately following diagnostic testing. To disable PPA debug output, delete the registry values under both the ..\Debug and ..\AMT keys.
Agent Logs - Introduction:
Agent Logs are the written interactions of the Symantec Management Agent (SMA) that is installed on client and server systems. The SMA is the software that establishes communication between the Notification Server (NS) on which the Symantec Management Platform (SMP) is installed and client computers. Please note that the NS on which the SMP is installed (SMP Server) by default also has the SMA installed as well. The reason for mentioning that detail is an attempt to alleviate any confusion when discussing client and server interactions because any server (including the SMP server) can also be a client. For the purposes of this document, client and server interaction will refer to the interaction between the SMP Server and another computer with the agent installed on it regardless of whether or not that client is a server system or a true client endpoint. Any computer with the SMA installed on it is referred to as a Managed Client.
Agent Logs - When to collect:
There are many reasons for collecting the agent logs. All of the solutions (SMP Plugin) available in the SMP use the SMA for communication. One reason for collecting them may be to determine which solution is causing the issue or if there are multiple problems. In general, the agent logs are a good starting point when diagnosing most problems in the SMP. Agent Logs are important to gather when having issues with a particular machine or a few specific machines.
Agent Logs - How to collect:
Client logs are located at C:\ProgramData\Symantec\Symantec Agent\Logs
Mac/Linux:
1.) Open a terminal.
2.) Enter the following command cd /opt/altiris/notification/nsagent/var.
3.) Enter the following command cat aex-client.log.
Server Logs - Introduction:
The Server Logs are essentially the inverse of the Agent Logs. Instead of detailing the interaction between a single client and the SMP, the Server Logs are the written interaction between the SMP and all Managed Clients. The Server Logs could be described as a one to many relationship where the Agent Logs can be described as a one to one relationship.
Server Logs - When to collect:
In most of the same instances that Agent Logs are gathered, it is also a good idea to gather the Server Logs. It is especially important to gather server logs when an issue is widely distributed amongst client computers rather than an issue with an individual computer.
Server Logs - How to collect:
Symantec Management Platform (Notification Server) logs are located at C:\ProgramData\Symantec\SMP\Logs
Verbose Agent Logs - Introduction:
Verbose Agent Logs are actually in the same location and are the same log files as the standard agent files, the only difference being that verbose logging has been enabled on those log files.
Verbose Agent Logs - When to collect:
Verbose Agent Logging is used in many of the same situations that standard agent logging is collected, but sometimes it is helpful to see every interaction between a particular client and the SMP Server. Verbose Agent Logs differ from Agent Logs by the amount of detail provided in the logs. Both the Symantec Management Agent and the server logging are controlled by the registry. To enable Verbose logging the registry will need to be edited.
Verbose Agent Logs - How to collect:
Please follow the steps below to collect the Verbose Agent Logs:
1.) Click Start > Run.
2.) Type Regedit and press Enter.
3.) Navigate to the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Altiris
\Altiris Agent\Event Logging\LogFile.
4.) Right click on the LogFile key and select New > DWORD (32-Bit) Value.
5.) Rename the new DWORD key to 'Severity'.
6.) Double click on the Severity key and give it a hexadecimal value of 'FF' and click OK.
7.) Right click on the LogFile key and select New > DWORD (32-Bit) Value.
8.) Rename the new DWORD key to 'MaxFiles'.
9.) Double click on the MaxFiles key and give it a decimal value of 100 (this increases the time
of living of the log files in minutes, the default is 1 or 2 minutes).
10.) After these registry changes have been made and the log files have been collected make
sure to delete the Severity and MaxFiles keys.
Verbose Server Logs - Introduction:
Verbose Server Logs are actually the same log files as the standard server logs. The only difference being that verbose logging has been enabled on those log files.
Verbose Server Logs - When to collect:
Verbose Server Logging is used in many of the same situations that standard server logging is collected, but sometimes it is helpful to see every interaction between all clients and the SMP Server. Verbose Server Logs differ from Agent Logs by the amount of detail provided within them. Both the Symantec Management Agent and the server logging are controlled by the registry. To enable Verbose logging the registry will need to be edited.
Verbose Server Logs - How to collect:
Please follow the steps below to collect the Verbose Agent Logs:
1.) Click Start > Run.
2.) Type Regedit and press Enter.
3.) Navigate to the following key:
HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\eXpress\Event Logging\LogFile.
4.) Right click on the LogFile key and select New > DWORD (32-Bit) Value.
5.) Rename the new DWORD key to 'Severity'.
6.) Double click on the Severity key and give it a hexadecimal value of 'FF' and click OK.
7.) Right click on the LogFile key and select New > DWORD (32-Bit) Value.
8.) Rename the new DWORD key to 'MaxFiles'.
9.) Double click on the MaxFiles key and give it a decimal value of 100 (this increases the time
of living of the log files in minutes, the default is 1 or 2 minutes).
10.) After these registry changes have been made and the log files have been collected make
sure to delete the Severity and MaxFiles keys.
Monitor Logs - Introduction:
To capture monitor logs a 3rd party tool like Microsoft's dbgview.exe must be used. To start Monitor Logging adjustments to the registry must be made.
Monitor Logs - When to collect:
The monitor logs should be collected when there is a suspected problem with the aexmetricprovider process. When monitor logging is enabled it collects verbose data about how the aexmetricprovider process operates.
Policy Data - Introduction: The client policy XML files contain monitor and other polices
Policy Data - When to collect:
Any time all policies applied to a particular system need to assessed it is a good idea to collect the client policy xml file.
Policy Data - How to collect:
The Client Policy XMLs are located at C:\Program Files\Altiris\Altiris Agent\Client Policies on all managed clients.
NT Event Logs - Introduction:
NT Event Logs are a component of Microsoft Windows that started with Microsoft Windows NT 4.0. They are a culmination of events from several system components
including: Application Event Logs, Security Event Logs, System Events Logs, Registry, and Message Resource files. The interface for viewing all of these logs
is through the Event Viewer (eventvwr.exe). The Event Viewer allows you to view, search, and record the NT Event logs. This discussion of NT Event Logs will be a bit
truncated here as there are entire books written on this subject matter.
NT Event Logs - When to collect:
NT Event Logs are important to collect when certain issues are occurring such as: monitor agent is crashing, Symantec Management Agent is crashing, Monitor Agent is causing
high CPU usage, WMI isn't working correctly, etc. In general if there is any question about what is causing the problem at hand, look through the NT Events to see if there
are any outstanding errors or warnings that could be related.
NT Event Logs - How to collect:
1.) Click Start > Run and type eventvwr and click OK.
2.) There are many options for viewing the Event Logs within Event Viewer, but generally speaking the logs of interest will be Application, Security, and
System. These are located underneath the Windows Logs directory on the left hand side.
3.) Once the desired logs have been identified they can be exported by right clicking the desired log and selecting "Save All Events As".
4.) This will bring up a prompt to name and save the file. Give it a name and save it. It will be saved as a .evtx file type.
To view this file at a later time it can be opened by simply double clicking the file and it will load into Event Viewer.
WMI Diagnostic Data, Testing, and Logging - Introduction:
WMI (Windows Management Instrumentation) is a core Windows management technology that can be used to gather system data and control system processes and services locally or remotely. There are 3 methods to collecting diagnostic information on WMI. The first portion is the actual diagnostic information that can be gathered by using Microsoft's WMIDiagnosis Tool (WMIDiag). The second part involves testing the WMI database installed on a machine both locally and remotely across the network which can be done using wbemtest. The instructions here on using WMIDiag will be brief as the full instructions on using the tool are included with the download (WMIDiag.doc). The third portion is WMI logging which requires changes
to the registry of the computer in question.
WMI Diagnostic Data and Logging - When to collect:
Anytime there is an issue or a suspected issue with WMI these logs should be collected. For instance if WMI metrics are not alerting as expected then WMI diagnostic data and logs should be collected.
WMI Diagnostic Data - How to collect:
WMI Testing - How to test locally and remotely:
How to test WMI locally:
1.) Login into the computer in question using an administrator login.
2.) Click Start > Run and type wbemtest in the Run prompt to launch WMI Tester. Press Connect.
3.) Type: root\cimv2 in Namespace textbox and click Connect.
4.) Click Query button and enter the following WQL query “select state, name from win32_service” and click Apply.
5.) If this returns the state of all win32 services then the query worked as expected.
How to test WMI remotely through wbemtest:
1.) On the Notification Server, click Start > Run and type webemtest in the Run prompt. Press Connect.
2.) Type: \\remote-machine\root\cimv2 in Namespace textbox. Also provide username and password in the respective text boxes. Click Connect.
3.) Click the Query button and enter Query “select state, name from win32_service” and click Apply.
How to test WMI remotely through PowerShell:
1.) Click Start > Run and type PowerShell and press enter.
2.) Type the following command: $Computer = "machineInQuestion" . Where machineInQuestion is the machine that needs it's WMI connectivity tested. Press Enter.
3.) Type the following command: Get-WmiObject -Namespace "root\cimv2" -Class Win32_Service -Impersonation 3 -Credential domainName\administrator -ComputerName $Computer.
Where domainName is the domain name of the local domain. Press Enter.
4.) This will bring up a prompt to enter an administrative username and password. Enter the appropriate credentials and press Enter.
5.) This will return all of the services running or not on the machine in question. To verify the results look at a service that is running on the machine in question and then
make sure the results of the PowerShell query also say that particular service is running.
WMI Logging - How to collect:
How to view WMI Events in Event Viewer:
1.) Log into the computer you would like to collect WMI logs from.
2.) Click Start > Run and type eventvwr and click Enter.
3.) Click the View menu and select Show Analytic and Debug Logs.
4.) In the tree directory viewer on the left navigate to Applications and Service Logs > Microsoft > Windows > WMI-Activity > Trace.
5.) In the tree directory viewer right click the Trace log and select Properties.
6.) Click the Enable Logging check box to start the WMI event tracing.
7.) To save the output of the WMI-Activity Trace log, right click the Trace log and select save all events as. Then give the file a name and click Save.
Process Monitor Logs - Introduction:
Process Monitor (ProcMon) is an advanced monitoring tool for Windows that shows Realtime file system, Registry and process/thread activity. This can be a very useful tool in many situations, especially if the source of problem at hand is hard to track down.
Process Monitor Logs - When to collect:
Process Monitor logs (PM Logs) are a good to collect when there are problems with the monitor agent such as: excessive CPU usage, excessive memory usage, monitor agent crashes, or monitor agent is causing a Blue Screen of Death (BSOD).
Process Monitor Logs - How to collect:
1.) Download Process Monitor from http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
2.) Create a folder named ProcMon on the root of the C drive. Extract the contents of the zipped folder to C:\ProcMon.
3.) Double click Procmon.exe to start the program.
4.) When Procmon.exe first launches it will automatically start logging. This can cause unnecessary logging and slow operation of the operating system.
To disable logging until the task at hand is ready to be run and logged by Procmon.exe click File > Capture Events.
5.) Once the task to be analyzed is ready to run click File > Capture Events to enable it.
6.) Once the task to be analyzed has been run and logged by Procmon.exe, save the log files by clicking File > Save and then enter the path in which the .PML file is to be saved and click OK.
SQL Server Error Logs and Trace - Introduction:
There are two primary methods for diagnosing SQL Server issues: SQL Server Error Logs and SQL Server Profiler Trace. The SQL Server error logs contains user defined events and certain system events that can be used to troubleshoot issues on the SQL Server. The Profiler trace is a built in program to Microsoft SQL Server Management Studio. The Profiler is used for monitoring an instance of the Database Engine or Analysis Services. The Profiler trace can provide information on the following issues: Stepping through problem queries to find the cause of the problem, finding and diagnosing slow-running queries, capturing the series of SQL transactions that can then be used on a test server to recreate the issue and diagnose it, monitor the performance of SQL Server to adjust workloads, correlating performance counters to diagnose problems.
SQL Server Error Logs and Trace - When to collect:
SQL Server error logs and profiler trace are a good first step for any potential database problem. For instance if monitoring reports are working as expected it may be a good idea to check the tables that the report is populated from and you can step through the SQL queries with the profiler or you can scan the error logs for potential problems that may be related.
SQL Server Error Logs - How to collect:
1.) Log on to the SQL Server with administrator credentials.
2.) Click Start > All Programs > Microsoft SQL Server Management Studio.
3.) Once Management Studio has started, click View > Object Explorer and this will bring up the Object Explorer view on the left hand side.
4.) In the Object Explorer, expand a server, expand Management, and then expand SQL Server Logs.
5.) Right Click a log and click View SQL Server Log. This will bring up the error logs in the Log File Viewer.
6.) To export and save the file click Export within the Log File Viewer, select a location and click Save. It will be saved as a .log file.
SQL Server Profiler Trace - How to collect:
1.) Log on to the SQL Server with administrator credentials.
2.) Click Start > All Programs > Microsoft SQL Server Management Studio.
3.) Once Management Studio has started, click Tools > SQL Server Profiler.
4.) Choose the server type by selecting either Database Engine or Analysis Services and click Connect.
5.) This will bring up the Trace Properties Window, give the trace a name and click Run.
6.) After the trace has started, it can be saved by clicking the red stop button and clicking File > Save As > Trace File. Give the file a name and a location and it will be saved as a .trc file.
Feedback
