Errors are returned trying to access the CA Performance Manager web site to log in
search cancel

Errors are returned trying to access the CA Performance Manager web site to log in

book

Article ID: 36198

calendar_today

Updated On:

Products

CA Infrastructure Management CA Performance Management - Usage and Administration

Issue/Introduction

Problem:

When trying to access the CA Performance Management web UI to log in, an error is observed. The error will state:

 

HTTP ERROR: 503 

Problem accessing /pc/redirector. Reason: 

Service Unavailable 

Powered by Jetty:// 

 

Cause:

When examining the systems log files, we found the following in the DMService.log file:

ERROR | WrapperSimpleAppMain | 2016-01-11 08:54:00,207 | com.ca.im.portal.api.common.services.impl.database.DBGlobalAttributes 

| Error refreshing attributes from database 

org.springframework.dao.DataAccessResourceFailureException: nested exception is java.net.ConnectException: Connection refused 

at com.ca.im.portal.api.common.services.impl.database.TransientDBConnection.processRecoverableDataAccessException(TransientDBConnection.java:100) 

 

Also seen in the systems log files, we found the following in the EMService.log file:

ERROR | pool-4-thread-1 | 2016-01-11 12:24:14,373 | com.ca.im.portal.em.ds.DeletedDataSourceRemover 

| Delete of data source failed. 

org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [DELETE from items WHERE sourceID=3]; SQL state [HY000]; error code [144]; Table './em/items' is marked as crashed and last (automatic?) repair failed; nested exception is java.sql.SQLException: Table './em/items' is marked as crashed and last (automatic?) repair failed 

at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83) 

 

Those log files are found in these locations, in a system that uses the default /opt installation directory. Change /opt as needed to specify your installation directory is other than the default was chosen.

  • DMService.log: /opt/CA/PerformanceCenter/DM/logs/
  • EMService.log: /opt/CA/PerformanceCenter/EM/logs/

 

These errors are indications that the netqosportal and/or em MySql databases that reside on the CA Performance Manager host have become corrupted.

Unfortunately these problems will not be resolved by a simple restart of services. In order to resolve this we'll need to resolve the corruption with MySql commands after shutting down the CA Performance Manager services, leaving the MySql service running.

 

Solution:

Notes:

  • If there is any concern or doubt about executing the below steps without assistance, please do not hesitate to open a new support case to request assistance.
  • None of the below processes require the MySql service to be stopped or restarted.
  • Before starting to fix the database, shut down all CA Performance Manager services to prevent DB access during the fix process. See KB article TEC1382101 for instructions on stopping and starting these services.
  • Run all MySql commands as the root user from the /opt/CA/MySql/bin directory.
  • Sometimes multiple executions of the myisamchk will be necessary to complete the recovery for all tables.
  • It is sometimes normal to see errors when some attempts to fix some tables are made. If that is seen try running the command again.
  • If persistent errors are seen for specific tables that aren't able to be fixed, open a new support case for further investigation.
  • Sometimes adding the --force flag to the myisamchk command run will help to fix stubborn tables. Run "myisamchk -h" for additonal help information.
  • Another trick for stubborn tables is to try fixing them by using the --safe-recover (-o), the --force (-f) option or by not using the --quick (-q) flag if its being used, when running the myisamchk command. 

 

After shutting down the four CA Performance Manager services, but leaving the MySql service running, the first step is to determine how extensive the problem is. To do so:

  1. Log into the CA Performance Manager server at the command line as the root user
  2. Change directories to the /opt/CA/MySql/bin directory.
  3. Run the following command to list the status of all tables in all databases present: mysqlcheck --all-databases 

Most common is to see the *.MYI named tables as corrupted in the output. Any tables without an OK will require resolution. If that is what we see the following command(s) will need to be run:

  • myisamchk --recover /opt/CA/MySql/data/netqosportal/*.MYI 
  • myisamchk --recover /opt/CA/MySql/data/em/*.MYI 

Change the path as needed if other than the default /opt install location was used.

 

When the "mysqlcheck --all-databases" command is run and all tables are returned as OK, the corruption has been resolved. It is now safe to restart the four CA Performance Manager services. After this the web UI and log in should work. If it does not, please open a new support case for further investigation and resolution.

Environment

Release: IMDAGG99000-2.5-Infrastructure Management-Data Aggregator
Component:
Powered by Wolken Software