Intermittent data from CA5620SamY1731 device pack

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

This NFMP is generating about 10K to 12K files an hour, however the device pack only seems to be processing about 8000 with a 15 minute poll interval, and we are seeing intermittent data on charts on some of the devices.

Also, I noticed the following message in one of the device pack logs:

Apr 27, 2021 11:30:23 AM, [[email protected]/Thread-7/168],  com.torokina.tim.plugin.sftp.TimSftpDownloader recursiveDownload
INFO: Reached maxBatchSize = 1000, exiting Sftp Plugin

Here are the parameters are currently set in CA5620SamY1731-SAS.xml

TimSftpFunction:     MaxDownloads=2000, and MaxBatchSize=1000
TimCopyFunction:     MaxDownloads=1000 

What is the correct way to modify the settings to be able to receive and process 12000 files per hour instead of 8000?

I bumped MaxBatchSize up to 3000 in CA5620SamY1731-SAS.xml, however, after doing that, I noticed that several devices were no long reporting data that had been before I made the change, so I rolled the change back.  
Next I tried bumping up MaxDownloads in both the TimSftpFunction com.torokina.tim.plugin.sftp.TimSftpFunction and the com.torokina.tim.plugin.copy.TimCopyFunction functions, and got the same result (no data on some devices), so I again rolled the change back.

Environment

All supported DX NetOps CAMM CA5620SamY1731 device pack releases

Cause

To many small files at once for the CAMM code to process at once.

Resolution

Engineering provided the following solution. It tells the code to combine the many small files into larger files to resolve the problem and keep up with the incoming rate of data.

1. Login to MC machine vi SSH
2. Copy the zip file /opt/CA/CAMM/MC/PM_devicepacks/CAAVIController/ENGINE_CAAVIController.zip to /tmp or a similar location to store it while working on it.
   • It is attached to this Knowledge Base article.
3. Unzip the ENGINE_CAAVIController.zip
4. In the script ENGINE_CAAVIController/repository/TIM-INF/groovy/scripts/scripts/MergeCAMMXMLs.groovy:
   • Replace the string "CAAVIController" with "CAAlcatel5620Sam"
5. Copy the file MergeCAMMXMLs.groovy that was edited to the MC machine, placing it in the directory:
   • /opt/CA/CAMM/MC/repository/<LC_IP>/COMPONENTS/ENGINE_CA5620SamY1731/TIM-INF/groovy/scripts/scripts/
6. Update the /opt/CA/CAMM/MC/repository/<LC_IP>/COMPONENTS/ENGINE_CA5620SamY1731/deviceConfig/CA5620SamY1731-SAS.xml file.
   1. Add the function:
      • scripts.MergeCAMMXMLs
   2. Refer to the ENGINE_CAAVIController\repository\deviceConfig\CAAVIController-Performance.xml file for the usage of function scripts.MergeCAMMXMLs.

Additional Information

The default section of the MergeCAMMXMLs.groovy script looks like:

		cammxmls.each {
			path->
				path.each{

After editing the lines now look like:

		cammxmls.each {
			//path->
				//path.each{