Symantec Management Agent (Altiris Agent) Troubleshooting

Products

• IT Management Suite
• Client Management Suite

Issue/Introduction

Troubleshooting information available to help with Symantec Management Agent 

Environment

ITMS 8.x

Resolution

Terminology:

SMA: Symantec Management Agent.
NS: Notification Server.
SMP: Symantec Management Platform.
SMP Server: Notification Server on which the Symantec Management Platform is installed.

 

The following new features were added to Symantec Management Agent (SMA) 7.5 to help troubleshoot:

1.    There is ability to redirect all OutputDebugStringA and OutputDebugStringW calls made by any SMA Plugin or DLL loaded in context of SMA Service into SAM log. Simply add greater than zero REG_DWORD "InterceptDebugOutput" value to 'HKLM\Software\\Altiris\\Altiris Agent\Diagnostics' registry key and restart SMA service. All the debug output will be redirected to the agent log marked by source "DebugOutput" and 'debug' severity.
2.    There is a limit of 15 seconds given SMA Service to gracefully stop. If some plugin or SMA component delays unloading this timeout may expire and SMA Service will terminate itself. Starting with build 7.5.1566 there is ability to create AeXNSAgent.exe's memory dump. Simply add REG_DWORD"dumpOnProcessTerminate" = 1 to 'HKLM\SOFTWARE\\Altiris\\Altiris Agent\\Diagnostics' registry key. There is no need to restart the service. Once the condition occurs the standard SMA EZ memory dump file will be created. The exception record for that dump will have the exception code 0x0000042B. The dump file may help to understand why a plugin or a component fail to shut down in the specified amount of time.
3.    There is an old registry value "Save Policy Requests" under ''HKLM\Software\\Altiris\\Altiris Agent\"registry key that forces SMA to dump the copy of HTTP policy requests and responses into "Altiris Agent\Client Policies" folder. A few adjustments were made because of Agent Trust Initiative implementation.

a.     If the value is equal to one (1) then the policy response will saved in a binary format to a file with DAT extension, the policy response header will be save to a text file with TXT extension. DAT file can be decrypted using SMATool.exe utility that can be found on SMP server in"Bin\Tools" folder. Use "SMATool.exe /data decrypt <file path>" command. SMATool.exe should run under SYSTEM account in order to be able to decrypt the policy response file (use psexec.exe).
b.    If the registry value is equal to two (2) then there will be a set of policy requests and response files created - each policy update will generate own uniquely named set of files in "Client Policies" folder, each file is marked by the date and time of policy request. This way you can see the policy changing over time on the agent.

4.    Push service install log AltirisAgentInstSvc.log has been combined with agent installer log AeXNSC.log into the single file. The combined log is named AEXNSC.log and located in the same folder where old AeXNSC.log lived. The install log now contains more information for troubleshooting purposes:

a.     Events are categorized into errors, warning, info and debug. Now all the modules that logs an entry into the agent log will log the same entry into the install log if module is running during install, upgrade or uninstall. By default only errors, warning and info entries are logged.
b.    Event source was added for better troubleshooting.
c.     Event format has NOT been changed to XML for better reading when no XML reader is available on the client machine.
d.    You can configure AeXNSC.exe to log the debug entries by specifying "-installlogging:debug" switch. Example: AeXNSC.exe -installlogging:debug
e.    You can configure AeXNSC.exe to turn the install log off by specifying "-noinstalllogging"switch
f.     You can configure push service AltirisAgentInstSvc.exe and AeXNSC.exe to log the debug entries by specifying "-installlogging:debug" switch in the additional parameters editbox when starting a push install.
g.    You can configure push service AltirisAgentInstSvc.exe and AeXNSC.exe to turn the logging off by specifying "-noinstalllogging" switch in the additional parameters editbox when starting a push install.

5.    AeXNSAgent.exe and AeXAgentUtil.exe got the new commands to help configuring agent log's severity, these commands can be also specified during push install or when installing from AeXNSC.exe:

a.     -enablelogging - configures agent to write error, warning and info entries in the agent log
b.    -enablelogging:error -  configures agent to write only errors in the agent log
example: AeXNSC.exe -enablelogging:error
c.     -enablelogging:warning - configures agent to write error and warning entries in the agent log
d.    -enablelogging:info - configures agent to write error, warning and info entries in the agent log
e.    -enablelogging:debug - configures agent to write error, warning, info and debug entries in the agent log.

6.    The ability to disable OLE Automation string cache has been added to simplify memory leaks troubleshooting. Add greater than zero REG_DWORD "disableOLEAutomationStringCache" value to 'HKLM\Software\\Altiris\\Altiris Agent\Diagnostics" registry key and restart SMA service or other SMA process. The process will disable OLE Automation cache upon restart so the cached strings are not reported as leaked by the various tools.
7.    Agent Storage troubleshooting:

a.     If some storage operation that can modify the storage content is unexpectedly interrupted, for example by a process termination of power drop, then storage remains marked as "dirty".
b.    By default SMA Service performs fast storage integrity check on startup, which validates only the storage's root integrity
c.     SMA service performs full storage integrity check on startup if storage is marked as dirty
d.    You can control "dirty" mark of the storage by running "SMATool.exe /storage dirty query"
e.    You can mark storage as dirty by running "SMATool.exe /storage dirty set"
f.     Only full integrity check can reset "dirty" mark
g.    "startupStorageIntegrityCheck" DWORD registry entry under 'Diagnostics' key controls how SMP agent service checks the storage on startup
 

0 - do not perform full integrity check
1 - (default), perform full integrity check if storage is marked as dirty
2 - always perform full integrity check 

h.    If storage's root structure is damaged, then full integrity check will be performed in any case.

 

The new features were added to SMA 7.6:

1.    There is a possibility now to automatically copy the dump files to some UNC location right after the creation of a dump file. The registry keys for regulation: "HKEY_LOCAL_MACHINE\SOFTWARE\Altiris Agent\Diagnostics". String values: "DumpRemotePath", "DumpRemoteUser", "DumpRemotePassword". User and password are in plain text and used for remote connection authentication. The keys should be created manually. By default they do not exist.
2.    The new commands have been added to SMATOOL.EXE:
 

a. /DATA DUMP LEGACYPASSSWORD - decrypts encrypted and base64 encoded legacy password string
b. /FILE DUMP POLICY <Policy DAT file> - decrypts encrypted policy file
c. /FILE DUMP STORAGE <Secure storage file> - decrypts a file from agent secure storage
d. /FILE DUMP PROFILE <Connection profile XML file> - decrypts encrypted connection profile file
 

3.    "Save Policy Requests" registry entry can be used to intercept connection profile requests made by the agent. The profile request file will have GUID as the first part of the file name, the GUID corresponds to the connection profile item GUID on the server.
4.    You can now add the diagnostics tools capabilities for the SMA by running from the command prompt "AeXNSAgent.exe /diags" or "AeXAgentUtil.exe /diags". It adds a new menu called "Diagnostics" in the Agent interface.

 

 

The new features were added to SMA 8.0:

1.   The new compressed and encrypted file format is introduced for the memory dump files the agent produces. The format is named EZ2, it provides authenticated recoverable encryption. The dump file can now be partially restored in case EZ2 file is damaged. The new utility called SMADumpDecoder.exe is shipped as part of SDK and SMP server installation. You should still use the older DumpDecoder.exe utility to decrypt EZ files.

2.   The log files are now embedded into the DMP file. By default, around 1MB of the latest agent log files get into the dump. This 1MB value can be changed by modifying DWORD registry value "embeddedLogFilesSoftLimit" in HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\Diagnostics.  The value sets the soft limit for the total size of the embedded log files in KB.
If value is zero then no log files will be embedded. if value is not zero then after embedding the file the agent will check if the total size of embedded log files exceeds the specified value, if it is then no more files will be embedded. SMADumpDecoder.exe has been modified to extract the embedded files from DMP files. If EZ2 file is passed as the input then utility will decrypt EZ2 file and create DMP file, then it will extract the embedded files from DMP file into the output folder. if DMP file is passed as the input then utility will extract the embedded files from DMP file into the output folder. Output paths in both cases are optional, the utility will form the output path itself if it is missing. The maximal hardcoded size of all the embedded log files is limited by 100MB. The logs files are collected in memory so in order to not overload the slow or low RAM machines in case user makes a mistake and specifies too large value in the registry the maximal value is limited by 100MB. The default 1MB should be enough in most of the cases. The maximal number of embedded files is limited by 10000.

3.   "DumpRemotePath" registry value can now contain the environment variable. You can use any system variable or custom variables set internally by SMA, these are:

SMA_VERSION - SMA agent version, like 8.0 if full version is 8.0.1234.5
SMA_BUILD_NUMBER - SMA agent build number, like 1234 if full version is 8.0.1234.5
SMA_REVISION - SMA agent build revision, like 5 if full version is 8.0.1234.5
WINDOWS_VERSION - Windows version, like 6.1
WINDOWS_REVISION - Windows revision, like "Service Pack 1"
WINDOWS_BUILD_NUMBER - Windows build number, like 1670
SMA_CRASH_PROCESS - the process that crashed, like "AeXNSAgent.exe"
SMA_CRASH_MODULE - the module that crashed, like ""ole32.dll"
Path sample \\10.31.8.74\REMOTE\%COMPUTERNAME%\%SMA_VERSION%\%PROCESSOR_ARCHITECTURE%\%WINDOWS_VERSION%\%WINDOWS_REVISION%\%WINDOWS_BUILD_NUMBER%

These folders hierarchy will be created automatically.

3.   SMATOOL utility is able to dump agent resource keys for the specific machine.  Run SMATOOL /AGENT DUMP RESOURCEKEYS command on the specific machine to see what info was used to generate the resource keys and the keys themselves.

Available options for AeXAgentUtil post- SMA 7.6:

AeXAgentUtil.exe [Optional Parameters]

Parameters:
/? | /h | /help Shows this help

/start Starts the Agent
/stop Stops the Agent
/restart Stops and starts the Agent
/recover Stops the Agent and restarts it only if it stopped without crashing

/sendbasicinventory Forces the Agent to send basic inventory
/updateconfiguration Forces the Agent to update configuration
/resetguid[:<delay sec>] Resets the Agent ID and forces the Agent to register on Notification Server after the optional delay
/registerguid[:<delay sec>] Forces the Agent to register on Notification Server after the optional delay

/diags Enables the Agent diagnostics
/nodiags Disables the Agent diagnostics

/uninstall | /clean Uninstalls the Agent
/uninstallagents Uninstalls all the installed plug-ins
/removepsfiles Removes the files downloaded by the Package Server plug-in
/removepackageserver Removes the files downloaded by the Package Server plug-in and uninstalls the Package Server plug-in
/registerclient Registers all the core agent modules

/nologging Disables the logging
/enablelogging:error Enables the error logging
/enablelogging:warning Enables the error and warning logging
/enablelogging:info Enables the error, warning and informational events logging
/enablelogging:debug Enables the error, warning, informational events and debug events logging
/enablelogging Enables the default logging

/server:<server> | /ns:<server> Switches the Agent to the new Notification Server specified by the host name <server>
/web:<web> | /nsweb:<web> Switches the Agent to the new Notification Server specified by the URL <web>

/enableasp Enables use of ASP pages in IIS
 

 

SMA Logging (Agent Log)

This is a log file of the SMA itself. It contains all the information SMA is configured to log during its work time.
This log file is "rotated" - means old files are being overwritten when number of files reach maximum defined value.

The following are the usual locations for the SMA logs:
XP/Server 2003:  
  
<drive:>\Program Files\Altiris\Altiris Agent\Logs.
Note: Please be advised that "C:\Program Files" could vary depending on SMA bitness vs System bitness. In case of 32-bit agents being installed on 64-bit OS-es, the path will start with "C:\Program Files (x86)".  

Vista/Windows 7/Server 2008:
  
   <drive:>\ProgramData\Symantec\Symantec Agent\Logs. 

Mac/Linux:
   1.) Open a terminal.
   2.) Enter the following command cd /opt/altiris/notification/nsagent/var.
   3.) Enter the following command cat aex-client.log. 

Verbose Agent Logs

The logging settings are stored in the following locations in the registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\Event Logging\LogFile

The following values are to be changed or created:

Value Name	Type	Default Value	Suggested Value	Description
MaxSize	DWORD (decimal)	100	2000	Size of an individual log file in kilobytes.
MaxFiles	DWORD (decimal)	20	20	Number of log files to keep/rotate.
Severity	DWORD (decimal)	7	255	Bitmask of levels for log messages to be written to the log.

Changing the settings above will cause Agent to write significantly more information to the logs. This is why it is necessary to increase the limits on the log file size.
Restarting the Agent is not required for the settings to take effect.

Please follow the steps below to collect the Verbose Agent Logs:
    1.) Click Start > Run.
    2.) Type Regedit and press Enter.
    3.) Navigate to the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\Event Logging\LogFile.
    4.) Right click on the LogFile key and select New > DWORD (32-Bit) Value.
    5.) Rename the new DWORD key to 'Severity'.
    6.) Double click on the Severity key and give it a hexadecimal value of 'FF' and click OK.
    7.) Right click on the LogFile key and select New > DWORD (32-Bit) Value.
    8.) Rename the new DWORD key to 'MaxFiles'.
    9.) Double click on the MaxFiles key and give it a decimal value of 100 (this increases the time of living of the log files in minutes, the default is 1 or 2 minutes).
    10.) Right click on the LogFile key and select New > DWORD (32-Bit) Value.
    11.) Rename the new DWORD key to 'MaxSize'.
    12.) Double click on the MaxFiles key and give it a decimal value of 2000 (this increases the size of living of the log files).
    13.) Restart the Symantec Management Agent service is not required in most cases for the changes can take effect.
    14.) After these registry changes have been made and the log files have been collected make sure to delete the Severity, MaxFiles and MaxSize keys.

Agent Install Service Logs

This is a log file created by installation service of an SMA named "AltirisAgentInstSvc.exe". This service is being installed while "push install" triggered on NS side or "pull install".
This log is being appended if already exist and contain information about how installation binary is being downloaded and launched, what installation parameters were transferred through command line to installation service.
Default log file location is:

Push:     C:\Windows\AltirisAgentInstSvc.log
            C:\Windows\system32\AltirisAgentInstSvc_del.log
Pull:      C:\Windows\system32\AltirisAgentInstSvc.log

Note 7.5 changes: this log does not exist anymore; it has been merged with Agent Installation Log below.

Install Agent Logs

This is a log file created by SMA installation. It covers all the install modes: install, upgrade, uninstall. Being re-created for each mode triggered, if added if already exist. Latest file is always "AEXNSC.log". The older files are being renamed to contain the date-time in their names like: "AEXNSC-20121122-182708.log".
This log contains detailed information about input parameters and installation environment data, what services are being stopped/installed, module registrations done and other auxiliary and useful information which could help to troubleshoot the installation process.

The following are the default location for the Install Agent Logs (after SMP 7.1 SP2 Release):
1. For machines Windows Vista and later, writes logs to:
"C:\ProgramData\Symantec\Symantec Agent\InstallLogs"
2. For Windows XP, or legacy agents prior to 7.1.7937:
"C:\Documents and Settings\All Users\Documents\Altiris\Altiris Agent\InstallLogs\"

SMA Crash Logs

The following are the default location for the crash agent dumps:
1. Windows Server 2008/2012
C:\ProgramData\Symantec\Symantec Agent\CrashDumps\
2. Windows Vista/Windows7/Windows 8:
C:\users\public\public documents\altiris\altiris agent 
3. Windows XP:
C:\Documents and Settings\All Users\Documents\Altiris\Altiris Agent
Note: The .ez files are a compressed format.  You need to use DumpDecoder to extract it and then you can use some of the other tools (such as Windbg) to read it afterwards.

You can find it under …\Program Files\Altiris\Notification Server\Bin\Tools

You basically run DumpDecoder.exe from the command prompt. You then you add the .ez dump file path and then the destination with the name that you want:

DumpDecoder.exe encoded_file_name out_file_name

Example:

C:\Users\Administrator.EPM>"C:\Program Files\Altiris\Notification Server\Bin\Tools\DumpDecoder.exe" C:\AeXNSAgentHostSurrogate32.dmp.ez C:\AeXNSAgentHostSurrogate32_extracted.dmp

Then, the extracted file should be a dmp file that you can run against a debugger tool.

Writing dump files may be disabled via registry, to enable set 'enableDumpCreation' to 1 in 'HKLM\SOFTWARE\Altiris\Altiris Agent\Diagnostics'.

 

 LDB folder:

The following are the default location for the LDB folder (Agent secure storage files):
"\Documents and Settings\All Users\Application Data\Symantec\Symantec Agent\Ldb"
"\ProgramData\Symantec\Symantec Agent\Ldb"

 

 Agent NSE Capture

To copy sent events into a specific folder on the Altiris Agent, you need to add the desired destination folder path in the existing [HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\Transport\Capture Events Folder] string value.

Notes:

• This setting will copy all events sent to the Notification Server. If for any reason the Agent cannot access its server the NS Events are left in the queue (path specified in the registry string [HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\Transport\Queue Path])
• The Symantec Management Agent doesn't need to be restarted for this setting to take effect.