uncategorized SQLException repeatedly in EM MOM log

book

Article ID: 207627

calendar_today

Updated On:

Products

CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We are getting this ERROR message over and over in our EM MOM log.

[ERROR] [pool-23-thread-4] [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.
java.lang.RuntimeException: org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [CREATE TEMPORARY TABLE IF NOT EXISTS appmap_vertex_filter  (vertex_id INTEGER, start_time TIMESTAMP, PRIMARY KEY (vertex_id, start_time) )  ON COMMIT DELETE ROWS;]; SQL state [XX000]; error code [0]; ERROR: right sibling's left-link doesn't match: block 34 links to 3844 instead of expected 3742 in index "pg_depend_depender_index"; nested exception is org.postgresql.util.PSQLException: ERROR: right sibling's left-link doesn't match: block 34 links to 3844 instead of expected 3742 in index "pg_depend_depender_index"
 at com.wily.introscope.appmap.em.services.impl.AGCServiceImpl.callFollowers(AGCServiceImpl.java:149)
 at com.wily.introscope.appmap.em.services.impl.AGCFollowerServiceImpl.callFollowers(AGCFollowerServiceImpl.java:100)
 at com.wily.introscope.appmap.em.services.ResponseWrapper.doRequest(ResponseWrapper.java:88)
 at com.wily.introscope.appmap.em.services.ResponseWrapper.doRequest(ResponseWrapper.java:83)
 at com.wily.introscope.appmap.rest.controllers.agc.AGCGraphController.getChangeEvents(AGCGraphController.java:833)
 at com.wily.introscope.appmap.rest.controllers.agc.AGCLandingControllerImpl.loadCacheRangeStatuses(AGCLandingControllerImpl.java:2192)
 at com.wily.introscope.appmap.rest.controllers.agc.AGCLandingControllerImpl.loadCache(AGCLandingControllerImpl.java:2257)
 at com.wily.introscope.appmap.rest.controllers.agc.AGCLandingControllerImpl.synchronizeCache(AGCLandingControllerImpl.java:2071)
 at sun.reflect.GeneratedMethodAccessor804.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
 at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [CREATE TEMPORARY TABLE IF NOT EXISTS appmap_vertex_filter  (vertex_id INTEGER, start_time TIMESTAMP, PRIMARY KEY (vertex_id, start_time) )  ON COMMIT DELETE ROWS;]; SQL state [XX000]; error code [0]; ERROR: right sibling's left-link doesn't match: block 34 links to 3844 instead of expected 3742 in index "pg_depend_depender_index"; nested exception is org.postgresql.util.PSQLException: ERROR: right sibling's left-link doesn't match: block 34 links to 3844 instead of expected 3742 in index "pg_depend_depender_index"
 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:90)
 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
 at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419)
 at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:445)
 at com.wily.introscope.appmap.em.model.dao.impl.gmt.GMTJdbcTemplate.execute(GMTJdbcTemplate.java:177)
 at com.wily.introscope.appmap.em.model.dao.impl.HelperOperationsDaoImpl.createTempTables(HelperOperationsDaoImpl.java:274)
 at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
 at com.sun.proxy.$Proxy123.createTempTables(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
 at com.sun.proxy.$Proxy124.createTempTables(Unknown Source)
 at com.wily.introscope.appmap.em.model.dao.impl.OperationsDaoImpl.setupFilter(OperationsDaoImpl.java:321)
 at com.wily.introscope.appmap.em.model.dao.impl.ViewOperationsDaoImpl.getChangeEvents(ViewOperationsDaoImpl.java:752)
 at sun.reflect.GeneratedMethodAccessor624.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
 at com.sun.proxy.$Proxy135.getChangeEvents(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor624.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at com.wily.introscope.appmap.em.model.dao.interceptor.ProfilingMethodInterceptor.invoke(ProfilingMethodInterceptor.java:54)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
 at com.sun.proxy.$Proxy136.getChangeEvents(Unknown Source)
 at com.wily.introscope.appmap.rest.controllers.GraphDBControllerImpl.getChangeEvents(GraphDBControllerImpl.java:250)
 at sun.reflect.GeneratedMethodAccessor743.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.wily.introscope.appmap.em.services.impl.AGCServiceImpl$FollowerInvocationHandler$1.run(AGCServiceImpl.java:478)
 at com.wily.introscope.appmap.em.services.impl.AGCServiceImpl.callFollowers(AGCServiceImpl.java:144)
 ... 19 more
Caused by: org.postgresql.util.PSQLException: ERROR: right sibling's left-link doesn't match: block 34 links to 3844 instead of expected 3742 in index "pg_depend_depender_index"
 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
 at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:310)
 at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:296)
 at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:273)
 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:268)
 at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:98)
 at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
 at org.springframework.jdbc.core.JdbcTemplate$1ExecuteStatementCallback.doInStatement(JdbcTemplate.java:436)
 at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408)
 ... 80 more

 

Environment

Release : 10.7.0

Component : APM Agents

Resolution

Customer re-indexed the table it was referring to (in this situation, "pg_depend_depender_index"), but the error continued to be reported.  When they re-indexed the entire database, both the CEM database and PostgreSQL databases, the error went away.