We cannot filter more than 635 CIs with this API call: https://xxx-server/rest/alert?size=635 meaning the error occurs with request size=636.
Error Message in logs :
2021-01-04 21:49:06,043 DEBUG [http-bio-0.0.0.0-7403-exec-11] exceptionmapper.WebApplicationExceptionMapper.toResponse(28) - org.jdom.IllegalDataException: The data "######## ELB_IBSK, Cannot connect to database instance ########\########, 1433: (connection timeout)." is not legal for a JDOM character content: 0x1f is not a legal XML character.
Release : 4.2
Component : Service Operations Insight (SOI) Manager
The issue is related to the data of an alert. There may be a special character coming in the message that is creating problem in parsing the alert by using REST API call.
To see that special character, you can copy and paste the SituationMessage and AlertDetail of the alert.
To fix current issue update the problematic alerts in SOI DB in alert manually as below:
a) Search for the error log as mentioned above in file CA\SOI\SamUI\logs\soiuis-debug.log
b) Find such problematic alarm by using the following query
select * from Alerts where active=1 and SituationMessage like '%Some_part_of_the_Alert_message_from_error_log%'
c) Update the SituationMessage and AlertDetail columns for that alarm by using following query
update alerts set AlertDetail='Remove_Special_Character_and_use_Correct alarm message_here' where alertid=alertid_of_the_problem_alarm
update alerts set SituationMessage='Remove_Special_Character_and_use_Correct alarm message_here' where alertid=alertid_of_the_problem_alarm
d) Stop and Start SOI services one by one
To avoid this issue to occur again the alerts should be corrected in the source product before publishing it to SOI.