ems probe not able fetch alarm details and API is returning server error (code 500)

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

We are seeing issues with EMS Probe where we are not able fetch alarm details and API is returning server error (code 500). it returns nothing sometimes and sometimes 500 error.

When we try to acknowledge bulk alarms it fails most of the time. For troubleshooting we are redeploying the probe again and it fixes all the issue. 

Resolution

There was something wrong with the ems probe as the ems.log wouldnt open with ERROR but now it looks better and responds as expected.

Steps Taken:

1. Adjusted the min memory to 2048 as the default value was very low.

2. Cleaned up a bunch of old error/pid files from 2024.

3. There is a known issue/small buglet in the logging (log4j2.xml) where the log size doesn't stop growing. The workaround was to change the log4j2.xml size and level. I've made those small changes. Previously, the log was building without rolling over.

Its working now after the changes and the ems log no longer grows without bounds.

log4j2.xml contents:

<?xml version="1.0" encoding="UTF-8"?>

<!--

<removed disclaimer>...

  -->

<Configuration>

    <Properties>

        <Property name="patternLayout">%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %X{client} %-5p %C:%M:%L %c{2} - %m%n</Property>

    </Properties>

    <Appenders>

   

        <!--  FILE: operatorconsolelogger -->

         <RollingFile name="ProbeLogFile" fileName="./ems.log" filePattern="./ems-%i.log">

            <PatternLayout pattern="${patternLayout}"/>

            <Policies>

                <SizeBasedTriggeringPolicy size="15 MB"/>

            </Policies>

            <DefaultRolloverStrategy max="5"/>

        </RollingFile>      

        

    </Appenders>




    <!-- Logger Levels: all, trace, debug, info, warn, error, fatal, off -->

    <Loggers>

         <Root level="error" additivity="false">

            <AppenderRef ref="ProbeLogFile"/>

        </Root>

        <Logger name="always" level="debug" additivity="false">

           <AppenderRef ref="ProbeLogFile"/>

        </Logger>




    </Loggers>

</Configuration>

 

4. Deactivated the ems probe

5. Deleted the db folder

6. Activated the ems probe

7. Tested again and no issues remained