UIM Maintenance Probe scheduler is not working

book

Article ID: 221871

calendar_today

Updated On:

Products

CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)

Issue/Introduction

 maintenance_mode probe isn't working at all, under it's log we can see:

 

Aug 10 12:39:08:113 WARN  [attach_socket, com.nimsoft.nimbus.NimServerSession] (120) Callback error, Exception in callback for public void com.nimsoft.monitor.probe.MaintenanceModeProbe.cbGetAllActiveWindowSchedules(com.nimsoft.nimbus.NimSession,int): PreparedStatementCallback; SQL [select mw.window_id, mw.schedule_id, mw.dev_id, mw.start_time, mw.end_time from maintenance_window mw, maintenance_schedule ms where ms.schedule_id = mw.schedule_id and mw.end_time > ? and mw.dev_id is null]; No more data to read from socket; nested exception is java.sql.SQLRecoverableException: No more data to read from socket: PreparedStatementCallback; SQL [select mw.window_id, mw.schedule_id, mw.dev_id, mw.start_time, mw.end_time from maintenance_window mw, maintenance_schedule ms where ms.schedule_id = mw.schedule_id and mw.end_time > ? and mw.dev_id is null]; No more data to read from socket; nested exception is java.sql.SQLRecoverableException: No more data to read from socket

               at com.nimsoft.nimbus.NimServerSession$NimServerSessionThread.performCallback(NimServerSession.java:233)

               at com.nimsoft.nimbus.NimServerSession$NimServerSessionThread.handleMessage(NimServerSession.java:178)

               at com.nimsoft.nimbus.NimServerSession$NimServerSessionThread.run(NimServerSession.java:123)

Caused by: org.springframework.dao.RecoverableDataAccessException: PreparedStatementCallback; SQL [select mw.window_id, mw.schedule_id, mw.dev_id, mw.start_time, mw.end_time from maintenance_window mw, maintenance_schedule ms where ms.schedule_id = mw.schedule_id and mw.end_time > ? and mw.dev_id is null]; No more data to read from socket; nested exception is java.sql.SQLRecoverableException: No more data to read from socket

               at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98)

               at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)

               at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)

               at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:660)

               at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695)

               at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:727)

               at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:737)

               at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:792)

               at com.nimsoft.nis.dao.MaintenanceWindowDao.query(MaintenanceWindowDao.java:176)

               at com.nimsoft.nis.dao.MaintenanceWindowDao.getAllActiveTimeWindows(MaintenanceWindowDao.java:160)

               at com.nimsoft.monitor.probe.util.MaintenanceWindowUtil.getAllActiveMaintenanceSchedules(MaintenanceWindowUtil.java:217)

               at com.nimsoft.monitor.probe.util.PDSBuilderForActiveWindows.buildPdsForActiveWindows(PDSBuilderForActiveWindows.java:40)

               at com.nimsoft.monitor.probe.MaintenanceModeProbe.cbGetAllActiveWindowSchedules(MaintenanceModeProbe.java:631)

               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

               at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

               at java.lang.reflect.Method.invoke(Method.java:498)

               at com.nimsoft.nimbus.NimServerSession$NimServerSessionThread.performCallback(NimServerSession.java:225)

               ... 2 more

Caused by: java.sql.SQLRecoverableException: No more data to read from socket

               at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:456)

               at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)

               at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)

               at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)

               at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)

               at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)

               at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:774)

               at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925)

               at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1111)

               at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798)

               at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4845)

               at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1501)

               at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)

               at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)

               at com.nimsoft.db.NimDBPreparedStatementOracle.executeQuery(NimDBPreparedStatementOracle.java:61)

               at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:703)

               at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:644)

               ... 16 more

 

Aug 10 12:40:00:405 ERROR [DefaultQuartzScheduler_Worker-7, com.nimsoft.monitor.probe.Notifier] Failed to notify nas /NTRG/PH1/JDCAPETSP2070/ems org.springframework.dao.RecoverableDataAccessException: StatementCallback; SQL [select ms.*,mw.WINDOW_START_TIME,mw.WINDOW_END_TIME, (select rtrim(xmlagg(XMLELEMENT(e,dev_id, ',').EXTRACT('//text()')).GetClobVal(),',') dev_ids from (select cd.dev_id from cm_device cd join maintenance_schedule_members msm on cd.cs_id=msm.cs_id where ms.SCHEDULE_ID = msm.SCHEDULE_ID AND cd.probe_name not like 'app_disco%')) dev_ids, (SELECT rtrim(xmlagg(XMLELEMENT(e,CS_ID||'='||DEV_ID, ',').EXTRACT('//text()')).GetClobVal(),',') Csids_devids FROM (select msm.CS_ID, cd.DEV_ID from cm_device cd join maintenance_schedule_members msm on cd.cs_id = msm.cs_id where ms.SCHEDULE_ID = msm.SCHEDULE_ID AND cd.probe_name not like 'app_disco%')) Csids_devids from MAINTENANCE_SCHEDULE ms left join (SELECT MAX(START_TIME) WINDOW_START_TIME,MAX(END_TIME) WINDOW_END_TIME, SCHEDULE_ID FROM MAINTENANCE_WINDOW group by SCHEDULE_ID) mw ON (mw.SCHEDULE_ID = ms.SCHEDULE_ID)]; No more data to read from socket; nested exception is java.sql.SQLRecoverableException: No more data to read from socket

Aug 10 12:40:00:407 ERROR [DefaultQuartzScheduler_Worker-7, com.nimsoft.monitor.probe.Notifier] Failed to notify nas /NTRG/PH1/JDCAPETSP2070/nas org.springframework.dao.RecoverableDataAccessException: StatementCallback; SQL [select ms.*,mw.WINDOW_START_TIME,mw.WINDOW_END_TIME, (select rtrim(xmlagg(XMLELEMENT(e,dev_id, ',').EXTRACT('//text()')).GetClobVal(),',') dev_ids from (select cd.dev_id from cm_device cd join maintenance_schedule_members msm on cd.cs_id=msm.cs_id where ms.SCHEDULE_ID = msm.SCHEDULE_ID AND cd.probe_name not like 'app_disco%')) dev_ids, (SELECT rtrim(xmlagg(XMLELEMENT(e,CS_ID||'='||DEV_ID, ',').EXTRACT('//text()')).GetClobVal(),',') Csids_devids FROM (select msm.CS_ID, cd.DEV_ID from cm_device cd join maintenance_schedule_members msm on cd.cs_id = msm.cs_id where ms.SCHEDULE_ID = msm.SCHEDULE_ID AND cd.probe_name not like 'app_disco%')) Csids_devids from MAINTENANCE_SCHEDULE ms left join (SELECT MAX(START_TIME) WINDOW_START_TIME,MAX(END_TIME) WINDOW_END_TIME, SCHEDULE_ID FROM MAINTENANCE_WINDOW group by SCHEDULE_ID) mw ON (mw.SCHEDULE_ID = ms.SCHEDULE_ID)]; No more data to read from socket; nested exception is java.sql.SQLRecoverableException: No more data to read from socket

Cause

java memory too low

Environment

Unified Infrastructure Management (Any version)

Maintenance_mode probe

Resolution

-Verify if error is displayed in the maintenance_mode log :ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT, if yes, work with Oracle DBA on the Oracle-related issue and fix increasing the Oracle PGA memory, to fix it. 

  • Check the ems and maintenance_mode and spectrumgtw probes, if the java memory min/max is too low so increase them to 2048/4096. And with spectrumgtw at 1024/2048.
  • Then a cold start (Deactivate-Activate) such probes.
  • Verify if no errors whatsoever seen in those probes any longer even at loglevel 5.

 

Additional Information

  • The status of a node in maintenance, e.g., white icon, should not be checked in IM as that is legacy behavior not related to OC for the state of the node.