APM 10.8 - ERROR: index "appmap_edges_pkey" contains corrupted page at block 0

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

APM teamcenter is not working properly, in the MOM log, we can see below ERROR message

6/20/23 05:21:45.996 PM CEST [ERROR] [pool-22-thread-5] [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.
org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [select count(1) from appmap_edges where fork in  (262144, 393216, 468992, 458752, 491520, 475136, 466944, 471040, 467968, 468480, 468736, 468608, 468672, 524288, 468687, 468686, 468684, 468680, 1048576, 468688, 468704)     AND start_time <= ? AND ? < end_time]; SQL state [XX002]; error code [0]; ERROR: index "appmap_edges_pkey" contains corrupted page at block 0
  Hint: Please REINDEX it.; nested exception is org.postgresql.util.PSQLException: ERROR: index "appmap_edges_pkey" contains corrupted page at block 0
  Hint: Please REINDEX it.
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:90)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:654)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:688)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:715)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:765)
        at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForObject(NamedParameterJdbcTemplate.java:211)
        at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForObject(NamedParameterJdbcTemplate.java:226)
        at com.wily.introscope.appmap.em.model.dao.impl.gmt.GMTNamedParameterJdbcTemplate.queryForObject(GMTNamedParameterJdbcTemplate.java:204)
        at com.wily.introscope.appmap.em.model.dao.impl.SupportabilityMetricOperationsDaoImpl.getCounts(SupportabilityMetricOperationsDaoImpl.java:44)
        at sun.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
        at com.sun.proxy.$Proxy200.getCounts(Unknown Source)
        at com.wily.introscope.appmap.supportability.DatabaseSupportabilityMetricsGatherer.updateCachedValues(DatabaseSupportabilityMetricsGatherer.java:101)
        at sun.reflect.GeneratedMethodAccessor335.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: ERROR: index "appmap_edges_pkey" contains corrupted page at block 0
  Hint: Please REINDEX it.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2297)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:322)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
        at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
        at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:696)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:638)
        ... 26 more

Environment

• Release: 10.x

Cause

The new error indicates the DB is corrupted, most likely due to some hardware failure.

Please check if there are any disk/file system issues.

https://wiki.postgresql.org/wiki/Corruption 

 

Resolution

Suggestions:

1. Try reindexing the corrupted index first:

    Reindex index <schema_name>.<index_name>;

2. If there are more corrupted indices, try reindexing the whole DB:

    reindex DATABASE <DATABASE NAME>;

However,  the chance of DB repairing/recovering is not very high, you should consider restoring a backup instead.