In the SEPM's Monitors, Logs screen: when trying to export a report with tens of thousands of records (computer status, for instance, for a large enterprise network over the past year), the SEPM seems to hang for several minutes and then ultimately fails. An "Internal Server Error" message, a "Bad Gateway" message, or other similar error is displayed instead of the report.
For several minutes, the SEPM seems to hang:
Eventually an Internet Explorer page opens with the title: "HTTP 500 Internal Server Error and body: The website cannot display the page." Clicking "More info" shows a message: "This error means that the website had a server problem which prevented the webpage from displaying."
In IE7, if the Tool->Internet Options->Advanced->Show friendly HTTP error messages checkbox is cleared, exporting the report opens the IE download dialog. The title of this dialog may read “101% of export_inventory.php from...” and an error message is shown says that the requested site is unavailable or cannot be found.
In IIS logs. POST export_inventory.php give 500 0 0 is seen.
In some cases, reports with fewer than expected number of records may be exported successfully. One sample Computer_status_export.txt (8.35 MB, 10233 lines long) concludes with:
It takes much longer for the SEPM to generate a report for exporting than it does to display the first page of that report on-screen. Even with the following changes, it can be normal when working with large data sets for a browser window to remain open for more than fifteen minutes as the exported file generates. (Much depends, of course, on the specifications of the server.) During this time, the php-cgi.exe will be seen in Task Manager to consume much CPU processing. If the desired report fails to be successfully created, it is typically the result of SEPM components "timing out." The errors referenced above are the result.
Several changes to configuration and maintenance steps are available which will offer improvements to performance. In many cases, the actions below will enable reports to generate successfully and swiftly.
To take advantage of recent performance enhancements and improvements, please ensure that the latest available release of SEP is installed on the SEPM. SEP 12.1 introduced many advances over the earlier SEP 11 product. The latest releases of either SEP 11 or SEP 12.1 are also likely to contain additional improvements over earlier builds.
In most cases, large reports can successfully be updated after implementing FastCGI, and then configuring increased timeout and memory limit values in the Php.ini and fcgiext.ini files.
Install FastCGI on SEP 11 SEPMs, and ensure that it is configured correctly.
FastCGI is a third-party tool from Microsoft which can improve the performance of the IIS component that the SEPM uses for Home, Monitors and Reporting. Refer the document "FastCGI_Setup_Readme.pdf" at "Tools\NoSupport\FastCGI\".
If FastCGI is not configured correctly, IIS will use CGI and set the default time out 300 seconds (5 minutes). This will often lead to time-outs when exporting very large reports.
Please note: the fcgiext.ini file is often not created automatically upon install. This may need to be manually created. A sample file:
Note: The sample fcgiext.ini file illustrated above uses the default 32-bit path for the SEPM install. This string will need to reflect the correct path, if the SEPM is installed on a 64-bit server or to a non-default location.
Configure ...\Symantec Endpoint Protection Manager\Php\Php.ini to Add "memory_limit = 512M" and change the max_execution_time timeout value
Modify the default php.ini file so that it can interact with FastCGI and have ample resources at its disposal.
Note: Be sure to reset IIS (iisreset) or restart the World Wide Web Publishing service after making the above changes!
Ensure that the SEM_GETUSN Stored Procedure is Enabled
This stored procedure was introduced in SEP 11 RU5. See the Improving SEPM Performance with the SEM_GETUSN Stored Procedure article for details on how to check if it is enabled in an environment.
If implementing FastCGI and configuing the .ini files has not enabled the SEPM to successfully export large repors, then additional advanced steps may be necessary. Please do not consider the following unless the above steps have already been attempted!
Changes Made on the SEPM
Change the MS SQL server connection timeout value and command timeout value
To change the Transaction timeout value
To change the IIS Connection timeout (SEP 11 only)
Change the CGITimeout Value Using Metbase Explorer from the IIS Resource Kit Tools (SEP 11 only)
Changes Made on the MS SQL 2005 Server
Defragment the SQL Server
Disk fragmentation can lead, over time, to poor performance by the MS SQL server. Running a Disk Defrag on the SQL Server's disk is recommended if SQL server performance is suspected in an SEPM issue.
Re-index the MS SQL 2005 Database
Be sure to create a full backup of the database before re-indexing or rebuilding.
Re-indexing can be done as part of a recommended maintenance schedule: for more information, see the article "Create database maintenance plans in MS SQL Server 2005 using SQL Server Integration Services (SSIS)."
The following script can be used to re-index the MS SQL database once:
Increase the size of the Tempdb
Tempdb is used by MS SQL servers for intermediate results of a query. It starts off at a small default size every time the server is restarted, and then automatically grows as necessary. There are a number of recommendtions for tuning the size of the file to increase perfromance. It is best to consult the following Microsoft technet document:
Working with tempdb in SQL Server 2005 http://technet.microsoft.com/en-us/library/cc966545.aspx
Change the Timeout Setting on the MS SQL Server
Changing from the default of 600 seconds to 1200 seconds and restarting the sql service will allow additional time for processing.
Symantec is continuing to explore additional changes and recommended practices. This document will be updated as soon as more information becomes available.