Symantec Endpoint Protection Manager Cannot Export Large Reports

book

Article ID: 177798

calendar_today

Updated On:

Products

Endpoint Protection

Issue/Introduction

In the SEPM's Monitors, Logs screen: when trying to export a report with tens of thousands of records (computer status, for instance, for a large enterprise network over the past year), the SEPM seems to hang for several minutes and then ultimately fails. An "Internal Server Error" message, a "Bad Gateway" message, or other similar error is displayed instead of the report.

Symptoms
For several minutes, the SEPM seems to hang:





Eventually an Internet Explorer page opens with the title: "HTTP 500 Internal Server Error and body: The website cannot display the page." Clicking "More info" shows a message: "This error means that the website had a server problem which prevented the webpage from displaying."

In IE7, if the Tool->Internet Options->Advanced->Show friendly HTTP error messages checkbox is cleared, exporting the report opens the IE download dialog. The title of this dialog may read “101% of export_inventory.php from...” and an error message is shown says that the requested site is unavailable or cannot be found.

In IIS logs. POST export_inventory.php give 500 0 0 is seen.

In some cases, reports with fewer than expected number of records may be exported successfully. One sample Computer_status_export.txt (8.35 MB, 10233 lines long) concludes with:




    Fatal error: Maximum execution time of 300 seconds exceeded in C:\Program Files\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Common\ado.php on line 108
    .

 

Cause

It takes much longer for the SEPM to generate a report for exporting than it does to display the first page of that report on-screen. Even with the following changes, it can be normal when working with large data sets for a browser window to remain open for more than fifteen minutes as the exported file generates. (Much depends, of course, on the specifications of the server.) During this time, the php-cgi.exe will be seen in Task Manager to consume much CPU processing. If the desired report fails to be successfully created, it is typically the result of SEPM components "timing out." The errors referenced above are the result.

Resolution

Several changes to configuration and maintenance steps are available which will offer improvements to performance. In many cases, the actions below will enable reports to generate successfully and swiftly.

First Steps

To take advantage of recent performance enhancements and improvements, please ensure that the latest available release of SEP is installed on the SEPM.  SEP 12.1 introduced many advances over the earlier SEP 11 product.  The latest releases of either SEP 11 or SEP 12.1 are also likely to contain additional improvements over earlier builds.

In most cases, large reports can successfully be updated after implementing FastCGI, and then configuring increased timeout and memory limit values in the Php.ini and fcgiext.ini files.

Install FastCGI on SEP 11 SEPMs, and ensure that it is configured correctly

FastCGI is a third-party tool from Microsoft which can improve the performance of the IIS component that the SEPM uses for Home, Monitors and Reporting. Refer the document "FastCGI_Setup_Readme.pdf" at "Tools\NoSupport\FastCGI\".

If FastCGI is not configured correctly, IIS will use CGI and set the default time out 300 seconds (5 minutes). This will often lead to time-outs when exporting very large reports.

Please note: the fcgiext.ini file is often not created automatically upon install. This may need to be manually created. A sample file:

    fcgiext.ini (at directory "C:\WINDOWS\system32\inetsrv\fcgiext.ini"):

    [Types]
    php=PHP

    [PHP]
    ExePath=C:\Program Files\Symantec\Symantec Endpoint Protection Manager\PHP\php-cgi.exe
    EnvironmentVars=PHP_FCGI_MAX_REQUESTS:10000
    InstanceMaxRequests=10000
    RequestTimeout=30000
    ActivityTimeout=30000


Note: The sample fcgiext.ini file illustrated above uses the default 32-bit path for the SEPM install. This string will need to reflect the correct path, if the SEPM is installed on a 64-bit server or to a non-default location.


Configure ...\Symantec Endpoint Protection Manager\Php\Php.ini to Add "memory_limit = 512M" and change the max_execution_time timeout value
Modify the default php.ini file so that it can interact with FastCGI and have ample resources at its disposal.

  1. Start Windows Explorer.
  2. Go to the following folder:

    Default location on SAV Reporter: C:\Program Files\Symantec\Reporting Server\PHP
    Default location on the SEPM: C:\Program Files\Symantec\Symantec Endpoint Protection Manager\Php
  3. Right-click the Php.ini file, and then click Properties.
  4. On the General tab, uncheck Read-only, and then click OK.
  5. Open the Php.ini file in a plain-text editor, such as Notepad.
  6. Find the memory_limit entry and increase the value to 512M.
  7. Find the max_execution_time entry and increase the value (in seconds).
    For example, to increase the timeout to ten minutes, change the line to the following:
    max_execution_time=600 (or higher, the sample php.ini below is set to 3000. Setting this value to 0 will allow unlimited time.)
  8. Save and close the Php.ini file.
  9. Right-click the Php.ini file, and then click Properties.
  10. On the General tab, check Read-only, and then click OK.

    An example:
    [PHP]
    cgi.force_redirect=0
    error_reporting=E_ALL & ~E_NOTICE
    extension=php_gd2.dll
    extension=php_mbstring.dll
    extension=php_mcrypt.dll
    extension_dir=.\ext
    max_execution_time=3000
    upload_max_filesize = 200M
    display_errors = Off
    display_startup_errors = Off
    log_errors = On
    report_memleaks = On
    track_errors = Off
    error_log = syslog
    expose_php=0
    allow_url_fopen=0
    session.gc_maxlifetime=2592000

    fastcgi.impersonate=1
    fastcgi.log=0
    memory_limit = 512M

    ;MBSTRING stuff
    mbstring.language = Neutral; Set default language to Neutral(UTF-8) (default)
    mbstring.internal_encoding = UTF-8 ; Set internal encoding to UTF-8
    mbstring.encoding_translation = On
    mbstring.http_input = UTF-8 ;
    mbstring.http_output = UTF-8 ; Set HTTP output encoding to UTF-8
    mbstring.detect_order = auto ; Set detect order to auto
    mbstring.substitute_character = none ; Do not print character
    output_buffering = On
    zlib.output_compression=On
    zlib.output_compression_level=6
    zlib.output_handler=mb_output_handler
    mbstring.func_overload = 7

    extension=php_openssl.dll
    extension=php_curl.dll
    session.save_path=C:\Documents and Settings\All Users\Application Data\Symantec\Symantec Endpoint Protection Manager\Php\temp\


Note: Be sure to reset IIS (iisreset) or restart the World Wide Web Publishing service after making the above changes!


Ensure that the SEM_GETUSN Stored Procedure is Enabled
This stored procedure was introduced in SEP 11 RU5. See the Improving SEPM Performance with the SEM_GETUSN Stored Procedure article for details on how to check if it is enabled in an environment.


Advanced Steps

If implementing FastCGI and configuing the .ini files has not enabled the SEPM to successfully export large repors, then additional advanced steps may be necessary. Please do not consider the following unless the above steps have already been attempted!

Changes Made on the SEPM

Change the MS SQL server connection timeout value and command timeout value

  1. Start Windows Explorer.
  2. Go to the following folder:

    Default location on SAV Reporter: C:\Program Files\Symantec\Reporting Server\Web\Resources
    Default location on the SEPM: C:\Program Files\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Resources
  3. Open the Reporter.php file in a plain-text editor, such as Notepad.
  4. Find the $CommandTimeout line and increase the value (in seconds).
    If the line does not exist, create it.
    For example, to increase the timeout to ten minutes, change the line to the following:

    $CommandTimeout = 600;
  5. Find the $ConnectionTimeout line and increase the value (in seconds).
    If the line does not exist, create it.
    For example, to increase the timeout to ten minutes, change the line to the following:

    $ConnectionTimeout = 600;
  6. Save and close the Reporter.php file.



To change the Transaction timeout value

  1. On the Windows taskbar, click Start > Programs > Administrative Tools > Component Services.
  2. In the left pane, expand Component Services > Computers.
  3. Right-click My Computer and then click Properties.
  4. On the Options tab, in the Transaction timeout box, type the value that you want (in seconds).
  5. Click OK.



To change the IIS Connection timeout (SEP 11 only)

  1. On the Windows taskbar, click Start > Run.
  2. In the Open box, type the following text:

    services.msc
  3. Click OK.
  4. In the right pane, right-click World Wide Web Publishing, and then click Stop.
  5. On the Windows taskbar, click Start > Programs > Administrative Tools > Internet Information Services.
  6. In the left pane, expand Web Sites.
  7. Right-click Symantec Web Server (or, if the Symantec components have been configured to run there, Default Web Site) and then click Properties.
  8. On the Web site tab, under Connections, in the Connection Timeout box, type the value that you want (in seconds).
  9. Click OK.
  10. On the Windows taskbar, click Start > Run.
  11. In the Open box, type the following text:

    services.msc
  12. Click OK.
  13. In the right pane, right-click World Wide Web Publishing, and then click Start.



Change the CGITimeout Value Using Metbase Explorer from the IIS Resource Kit Tools (SEP 11 only)

  1. Download the IIS 6.0 Resource Kit Tools from Microsoft and install these to the SEPM.
  2. Open the Metabase Explorer.
  3. Navigate to the CGITimeout value (LM-> W3SVC-> CGITimeout)
  4. Increase this value
  5. Close Metabase Explorer and restart the WWW service, as per above.




Changes Made on the MS SQL 2005 Server

Defragment the SQL Server
Disk fragmentation can lead, over time, to poor performance by the MS SQL server. Running a Disk Defrag on the SQL Server's disk is recommended if SQL server performance is suspected in an SEPM issue.


Re-index the MS SQL 2005 Database
Be sure to create a full backup of the database before re-indexing or rebuilding.

Re-indexing can be done as part of a recommended maintenance schedule: for more information, see the article "Create database maintenance plans in MS SQL Server 2005 using SQL Server Integration Services (SSIS)."

The following script can be used to re-index the MS SQL database once:

      use sem5

      exec sp_msforeachtable
      @command1="print '?' dbcc dbreindex('?', ' ', 80)"

      exec sp_updatestats



Increase the size of the Tempdb
Tempdb is used by MS SQL servers for intermediate results of a query. It starts off at a small default size every time the server is restarted, and then automatically grows as necessary. There are a number of recommendtions for tuning the size of the file to increase perfromance. It is best to consult the following Microsoft technet document:

Working with tempdb in SQL Server 2005 http://technet.microsoft.com/en-us/library/cc966545.aspx


Change the Timeout Setting on the MS SQL Server
Changing from the default of 600 seconds to 1200 seconds and restarting the sql service will allow additional time for processing.


Final note

Symantec is continuing to explore additional changes and recommended practices. This document will be updated as soon as more information becomes available.


Attachments