APM teamcenter is not working properly, in the MOM log, we can see below ERROR message
6/20/23 05:21:45.996 PM CEST [ERROR] [pool-22-thread-5] [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.
org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [select count(1) from appmap_edges where fork in (262144, 393216, 468992, 458752, 491520, 475136, 466944, 471040, 467968, 468480, 468736, 468608, 468672, 524288, 468687, 468686, 468684, 468680, 1048576, 468688, 468704) AND start_time <= ? AND ? < end_time]; SQL state [XX002]; error code [0]; ERROR: index "appmap_edges_pkey" contains corrupted page at block 0
Hint: Please REINDEX it.; nested exception is org.postgresql.util.PSQLException: ERROR: index "appmap_edges_pkey" contains corrupted page at block 0
Hint: Please REINDEX it.
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:90)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:654)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:688)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:715)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:765)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForObject(NamedParameterJdbcTemplate.java:211)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForObject(NamedParameterJdbcTemplate.java:226)
at com.wily.introscope.appmap.em.model.dao.impl.gmt.GMTNamedParameterJdbcTemplate.queryForObject(GMTNamedParameterJdbcTemplate.java:204)
at com.wily.introscope.appmap.em.model.dao.impl.SupportabilityMetricOperationsDaoImpl.getCounts(SupportabilityMetricOperationsDaoImpl.java:44)
at sun.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy200.getCounts(Unknown Source)
at com.wily.introscope.appmap.supportability.DatabaseSupportabilityMetricsGatherer.updateCachedValues(DatabaseSupportabilityMetricsGatherer.java:101)
at sun.reflect.GeneratedMethodAccessor335.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: ERROR: index "appmap_edges_pkey" contains corrupted page at block 0
Hint: Please REINDEX it.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2297)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:322)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:696)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:638)
... 26 more
The new error indicates the DB is corrupted, most likely due to some hardware failure.
Please check if there are any disk/file system issues.
https://wiki.postgresql.org/wiki/Corruption
Suggestions:
1. Try reindexing the corrupted index first:
Reindex index <schema_name>.<index_name>;
2. If there are more corrupted indices, try reindexing the whole DB:
reindex DATABASE <DATABASE NAME>;
However, the chance of DB repairing/recovering is not very high, you should consider restoring a backup instead.