DX UIM hammering Spectrum with schedule model creations/deletions

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• Unified Infrastructure Management for Mainframe
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)
• CA Spectrum
• DX NetOps

Issue/Introduction

UIM integration through the DX UIM spectrumgtw probe is impacting Spectrum performance severely. It is causing memory issues due to queuing, and causing the SpectroSERVER process to run at 90 to 100% utilization due to an execessive number of schedule creations and deletions.

 

It seems that the creations and deletions are mostly maintenance mode schedules, in a 24-hour period, at least 4 million creations and deletions occurred. The number is likely higher, but Spectrum is unable to keep up with logging to the database because of the sheer number of events hitting Spectrum.

Environment

• UIM Release: 8.51 SP1
• Spectrum: 10.3.0
• Components:
•    UIM - SPECTRUMGTW: 8.67HF1 and 8.67HF2
•    UIM - MAINTENANCE_MODE:  8.53HF3
•    UIM - SPECTRUM UIM SERVER:  8.67

Cause

In this instance, all maintenance schedules are created in Spectrum. Spectrum schedule names are automatically created by Spectrum. In some instances, the schedule names exceed 100 characters.

The UIM MAINTENANCE_SCHEDULE table currently has a hard limit of 100 characters for the SCHEDULE_NAME column. When the spectrumgtw probe requests the addition of the Spectrum maintenance schedule with a name that contains more than 100 characters, the schedule addition request fails. Errors similar to the following can be found in the maintenance_mode.log file at loglevel 5:

ERROR [attach_socket, com.nimsoft.monitor.probe.MaintenanceModeProbe] Failure adding schedule Every week on Sun, Mon, Tue, Wed, Thu, Fri and Sat  from 11:38 AM for 23 hours 59 min, starting May 27, 2019.org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL []; String or binary data would be truncated.; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: String or binary data would be truncated.

In this case the schedule name shown below:

Every week on Sun, Mon, Tue, Wed, Thu, Fri and Sat  from 11:38 AM for 23 hours 59 min, starting May 27, 2019

exceeds the 100 character limit!

This particular schedule was created by defining a Weekly schedule where every day of the week was checked off:

The spectrumgtw probe detects that the new schedule was not added to the MAINTENANCE_SCHEDULE table and initiates an immediate maintenance schedule sync which will continue to fail as long as the number of characters in the schedule name exceeds 100 characters.

In this instance, there were several hundred maintenance schedules already defined in the UIM MAINTENANCE_SCHEDULE table that were expired.

During the maintenance schedule sync performed by the spectrumgtw probe, requests are made to add missing UIM schedules to Spectrum in addition to adding missing Spectrum schedules to UIM.  The SpectroSERVER adds the missing schedule models, but since they are expired, the SpectroSERVER then immediately deletes the expired model.   

Since the maintenance sync was repeatedly running in the spectrumgtw probe because of the failed additions of the missing Spectrum schedules, this was flooding the SpectroSERVER with an excessive number of requests to repeatedly add the expired schedules from UIM.

Resolution

The following steps were performed to resolve this issue as a workaround:

1.  Deactivate the following set of probes:

• spectrumgtw
• maintenance_mode
• wasp probe on all UMP robots

2.  Reset the UIM MAINTENANCE tables by executing the following set of queries:

     TRUNCATE TABLE MAINTENANCE_WINDOW;

     TRUNCATE TABLE MAINTENANCE_SCHEDULE_MEMBERS;

     DELETE FROM MAINTENANCE_SCHEDULE;

3.  In Spectrum, create new schedules to replace those with schedule names > 100 characters.

     For example, instead of creating weekly schedules with every day checked, create a daily schedule that applies to every day of the week:

This creates a maintenance schedule with the name:

Every day from 11:38 AM for 23 hours 59 min

which contains less than 100 characters.

Associate the devices from the old maintenance schedules to the new maintenance schedules, then delete the problematic maintenance schedules.

4.  It may be necessary to restart the SpectroServer if it is still consuming too many system resources (CPU and/or memory).

5.  Delete the following files from the <installPath>\Nimsoft\probes\gateway\spectrumgtw\cache directory:

• SpectrumScheduleCache.dat
• UimScheduleCache.dat

6.  Activate the spectrumgtw probe. 

     This will re-sync all Spectrum maintenance schedules that are active which have devices associated with them in UIM.

Additional Information

Until a permanent solution is estabished for handling Spectrum maintenance schedules with schedule names that exceed 100 characters, it is recommended that care be taken with the creating of Spectrum maintenance schedules to assure that the schedule name does not contain more than 100 characters.

If this is not possible, then create the schedule from the OC maintenance webapp/UMP USM portlet which should restrict the size of the maintenance schedule name and associate devices to the new schedule.  The next time that the spectrumgtw probe syncs maintenanace schedules, a new schedule will be created in Spectrum.