System freezes up at same time every day | HistoryCompressorTask/TIMER taking a long time

Products

• CA Automic One Automation

Issue/Introduction

Early 12.x system seems to lock up at the same time every day.  The following symptoms are seen:

Java UI shows a pop up indicating that it cannot find a WP process
The environment seems to freeze up
After a few minutes, the environment returns to normal
Checking Usage in System Overview or Administration Perspective shows a an unexplained system load with a huge spike attributed to client 0

The JWP logs show something like this:

20210802/201156.967 - 24     U00003524 UCUDB: ===> Time critical DB call!       OPC: 'EXEC' time: '1213ms'
20210802/201156.967 - 24     U00003525 UCUDB: ===> 'DELETE FROM PMIAM WHERE PMIAM_PMMA_IDNR IN (SELECT PMMA_IDNR FROM PMMA WHERE PMMA_PMMD_Idnr = ? AND PMMA_Timestamp > ? AND PMMA_Timestamp <= ? )'
20210802/201724.190 - 24     U00003524 UCUDB: ===> Time critical DB call!       OPC: 'EXEC' time: '327223ms'
20210802/201724.190 - 24     U00003525 UCUDB: ===> 'DELETE FROM PMMA WHERE PMMA_PMMD_Idnr = ? AND PMMA_Timestamp > ? AND PMMA_Timestamp <= ? '
20210802/201725.772 - 24     U00003524 UCUDB: ===> Time critical DB call!       OPC: 'EXEC' time: '1581ms'
20210802/201725.773 - 24     U00003525 UCUDB: ===> 'DELETE FROM PMMAV WHERE PMMAV_PMMA_Idnr IN (SELECT PMMA_IDNR FROM PMMA WHERE PMMA_PMMD_Idnr = ? AND PMMA_Timestamp > ? AND PMMA_Timestamp <= ? )'
20210802/201757.582 - 24     U00003524 UCUDB: ===> Time critical DB call!       OPC: 'EXEC' time: '31358ms'
20210802/201757.582 - 24     U00003525 UCUDB: ===> 'DELETE FROM PMMA WHERE PMMA_PMMD_Idnr = ? AND PMMA_Timestamp > ? AND PMMA_Timestamp <= ? '
20210802/201757.595 - 23     U00003524 UCUDB: ===> Time critical DB call!       OPC: 'SLCT' time: '764406ms'
20210802/201757.595 - 23     U00003525 UCUDB: ===> 'SELECT PMMAV_Key, PMMAV_Value FROM PMMAV WHERE PMMAV_PMMA_Idnr = ? '
20210802/201757.636 - 24     U00003434 Server routine  'HistoryCompressorTask/TIMER' required '17' minutes and '57' seconds for processing.

The rest of the (regular) WP logs show:

10 - 20210803/201845.266 - U00003434 Server routine  'JPEXEC_R/TIMER' required '14' minutes and '29,449' seconds for processing.
13 - 20210803/201845.267 - U00003434 Server routine  'JPEXEC_R/TIMER' required '14' minutes and '45,661' seconds for processing.
15 - 20210803/201845.267 - U00003434 Server routine  'JPEXEC_R/TIMER' required '14' minutes and '27,032' seconds for processing.
14 - 20210803/201845.268 - U00003434 Server routine  'JPEXEC_R/TIMER' required '14' minutes and '32,211' seconds for processing.
02 - 20210803/201845.280 - 23     U00003434 Server routine  'HistoryCompressorTask/TIMER' required '18' minutes and '44' seconds for processing.

Environment

Release : 12.1

Component :

Cause

This issue is caused by the PMMA tables and the "PERFORMANCE" settings being turned on.  These settings are not widely used and can be turned off.

Resolution

The PMMA processes have been updated a lot since 12.1.x, but the way to stop this from happening with your system is by turning off the performance tracking:
1) Go into client 0
2) Open UC_SYSTEM_SETTINGS
3) Change the Key PERFORMANCE to N (or add it in case it wasn't there to begin with)
4) Change the Key PERFORMANCE_KEEP_DAYS to 9999 (or add it in case it wasn't there to begin with)

No restart is required when updating these.