snmpcollector probe scalability, performance tuning and optimization guidelines
Article ID: 9879
Updated On:
Products
Issue/Introduction
snmpcollector probe - performance tuning and optimization guidelines, as well as some troubleshooting tips.
- Scalability
- Performance tuning
- Optimization guidelines
- Troubleshooting tips
- Associated database queries
Environment
- DX UIM 23.4.* and above
- snmpcollector version 4.x or higher
Cause
- Guidance for improving snmpcollector scalability and performance
- Performance optimization
- Workload control
Resolution
The snmpcollector probe uses a large amount of memory and disk resources. For the best performance and scalability:
- Installation
Install the snmpcollector probe on a remote hub. Do NOT install it on a robot and do NOT install it on the Primary hub.
The hub that you use for SNMP data monitoring can have a significant impact on the performance and scalability of the probe. snmpcollector should be run from a hub because the spooler probe process is multi-threaded on a hub and a hub machine is usually allocated more resources. - Message Queues
Use manually configured hub queues to get the snmpcollector data back to the Primary hub. (Configure local ATTACH queue(s) on the remote hub, and configure GET queue(s) on upstream hubs/the Primary hub) - Processor speed
Ideally, use a machine that has processor speed of 3Ghz or higher with at least 4 virtual CPUs. - Memory and Disk
Allocate sufficient memory and disk space to support your device data collection requirements. The monitoring capabilities of the snmpcollector probe are limited if you install the probe on a small-medium scale system with a limited amount of resources. For more information about the minimum hardware requirements, see snmpcollector Hardware Requirements.
Only install snmpcollector with other monitoring probes if the other probes do not consume a large amount of system resources, e.g., such as storage probes. For example, you can install the cdm and ntservices probes if the hub has sufficient resources.
Sometimes it is necessary to increase the java heap memory to the probe. Open the probe in Raw configure mode and select startup->opt and start with at least 2048/4096 respectively but that is a minimum.
Important Limit Rule: Broadcom recommends keeping your Java minimum (-Xms) and maximum (-Xmx) settings within 2 or highest of 4 GB of each other (e.g., -Xms12g and -Xmx16g) to keep garbage collection performance predictable.
Applying Java Memory Settings
- Deactivate the snmpcopllector probe
- Wait for the port and PID to clear
- Adjust the java min/max settings
- Activate the probe
Host Hardware
In larger environments, if assigning a 16 GB heap to the probe, make sure the underlying VM or physical server has at least 32 GB of system RAM and a 64-bit multi-core processor (4 vCPUs) to leave breathing room for the operating system and the robot spooler.
| Scale / Environment Size | Total Components / Metrics Monitored | Recommended Java Memory (-Xms / -Xmx) |
Out-of-the-Box / Default | Baseline / Standard Testing / Test-Dev | 2.5 GB to 4 GB |
Medium Environments | Up to ~100,000 components | 4 GB to 8 GB |
Large Environments | 100,000 to 500,000 components | 10 GB to 12 GB |
Extra Large / High-Scale Environments | Greater than 500,000 components | 14 GB to 16 GB (or scale out) |
5. Scalability and the "Scale-Out" Rule
If your target environment exceeds 500,000 metrics or components, do NOT continue pushing the Java heap higher on a single probe. Instead, deploy a second/additional instance of snmpcollector on an additional remote hub and distribute/split the monitoring targets evenly across both instances.
- We do NOT recommend installing snmpcollector on a hub with other probes that can also consume a large amount of system resources.
Some examples of these probes are vmware, nutanix, icmp, ibmvm and any storage probes. Use filters as much as possible rather than creating multiple templates. Filters let you control how the probe applies monitors based on attributes of the target device.
IMPORTANT: every template that you create is read separately by the probe. The probe uses a large amount of system resources to read EACH template.
If you don't have the required system and available resources in your environment, you might see a slowdown in data processing or failures/gaps in QOS collection. - snmpcollector scalability/limitations
a. There is a max limit of 500000 components (interfaces, cpus, memory, disk, etc., per single snmpcollector probe instance. You must remain under the maximum limits of the snmpcollector probe - so if you surpass it on a single instance, at some point you may have to split up the monitoring across two instances.
b. Note that the url, http://localhost:9715 displays the current SNMPC component/device/metrics information. SNMPCollector web console: from v4.06, by default, a direct connection to the H2 database using SNMPCollector H2 console (port 9715) is disabled. To enable direct connection, use the probe’s configuration key enable_database_servlet = true the probes' setup section. The setting can be temporarily enabled to see the number of components.
c. You can increase the snmpcollector java probe's min/max memory to 14 and 16 GB or ideally deploy another instance of the snmpcollector probe on another hub with sufficient resources/memory and 'split up' the monitoring across the two instances. This is the best practice.
For one customer, the device components count hit as high as 731510 components on 701 devices with 254038 metrics enabled. In such cases, the queries listed below will help in making an assessment.
IMPORTANT!
When the limit of 500000 components is surpassed you may see symptoms occur such as the probe turns red and wont start, or you cannot open the probe in the Admin Console, and it generates a PPM-023 error, or gaps in monitoring, or there may be other unforeseen anomalies like the hub becoming unstable due to the increased workload on the hub where snmpcollector is deployed. Its not worth waiting until you reach or surpass the max.
- We do NOT recommend installing snmpcollector on a hub with other probes that can also consume a large amount of system resources.
Helpful Queries for Analysis (Scalability and Performance issues)
-- Number of configuration items per probe instance
select count(CI.CI_ID) as Comp_count,CD.dev_src_id, CNR.robot, CNP.port, CNP.active, CNP.pkg_version
from CM_CONFIGURATION_ITEM CI inner join CM_DEVICE CD on CD.dev_id = CI.dev_id
inner join CM_NIMBUS_ROBOT CNR on CD.dev_src_id = CNR.dev_id
inner join CM_NIMBUS_PROBE CNP on CNP.robot_id = CNR.robot_id
where CNP.probe_name = 'snmpcollector'
group by CD.dev_src_id, CNR.robot, CNP.port, CNP.active, CNP.pkg_version
order by Comp_count DESC;
-- Number of configuration items per snmp device showing device name and count
select cd.dev_name, COUNT(*) as count from CM_CONFIGURATION_ITEM ci
inner join CM_DEVICE cd on ci.dev_id=cd.dev_id
where cd.probe_name='snmpcollector'
group by cd.dev_name order by count desc;
-- Number of configuration items per snmp device showing where its monitored from, the device name and count
select cr.robot as 'Monitored From', cd.dev_name, COUNT(*) as count from CM_CONFIGURATION_ITEM ci inner join CM_DEVICE cd on ci.dev_id=cd.dev_id
inner join CM_NIMBUS_ROBOT cr on cr.dev_id=cd.dev_src_id
where cd.probe_name='snmpcollector' group by cr.robot, cd.dev_name order by cr.robot
Additional H2 database query
Navigate to http://<snmpcollector_ip>:9715/database in your web browser.
Note: Direct connection to the H2 database console on port 9715 is disabled by default in newer versions (v4.06+). You must temporarily enable it by setting the configuration key enable_database_servlet to true under the Setup section of the snmpcollector probe configuration.
On the authentication login screen, click Connect without altering the default parameters.
Then click run to execute the specific query Broadcom support has provided and requested you to run.
Warning!
Broadcom strongly discourages modifying or writing update/delete queries directly to this relational database unless explicitly directed by Broadcom support as it can corrupt internal metric associations and profile mappings.
-- H2 database query to assist in distributing workload across other snmpc instances
-- snmpcollector instances with components over the 500k limit
-- The query below identifies devices with the highest number of components.
-- Please review the results and consider moving the devices with the highest component counts to other/additional SNMPC instances
-- or deleting them if they are no longer required.
SELECT s.hostname, count(*) AS count
FROM component c
INNER JOIN snmpdevice s ON s.deviceid = c.deviceid
GROUP BY s.hostname
ORDER BY count DESC;
Other Considerations
- ppm probe
Check to make sure that the ppm probe version matches the current installed version of UIM. - Required Metrics/KPIs
In terms of what you plan on monitoring, please consider what metrics provide actual value for your environment/what metrics customers will actually need to see and/or use in reports/dashboards. A sound strategy would include best practices/KPIs for given devices, e.g., routers, switches etc.
For example:
Routers and Switches:
- Availability
- CPU
- Memory Utilization
- Buffer Hit Stats
Load Balancers:
- CPU
- Memory
- Availability and Responsiveness
- Concurrent Sessions
- Connections per seconds
3. Polling (monitoring) intervals
- Symptoms of a polling interval that may be too short might include when polling is taking > 5 minutes when the interval is already set to 5 minutes, and it causes the data points to be more than 5 minutes apart.
4. Protocols - other performance considerations (Use of SNMPv1 versus SNMPv2/v2c)
What is SNMP v1?
SNMP v1 (also known as SNMPv1 or SNMP version 1) is the initial version of SNMP protocol. SNMP v1 is defined in RFC 1065 to 1067 and 1155 to 1157. It was developed by a small group of collaborators at a time when Internet standards and security were not paid much attention. SNMP v1 operates over UDP (User Datagram Protocol), IP (Internet Protocol), CLNS (OSI Connectionless Network Service), DDP (AppleTalk Datagram-Delivery Protocol) and IPX (Novell Internet Packet Exchange). SNMP v1 uses the authentication mechanism of transmitting a “community string” (i.e. a password) in clear text, which is very insecure.
What is SNMP v2?
SNMP v2 (also known as SNMPv2 or SNMP version 2) is defined in RFC 1441 to RFC 1452. SNMP v2 adds several improvements over SNMP version 1. They are improvements in performance along with advancements in security and confidentiality. It also adds improvements in the area of manager-to-manager communication. GetBulkRequest has been added to retrieve large data amounts by a single request. Earlier, you had to use GetNextRequest iteratively in order to get a bulk of data. However, many users believed that the party-based security system in SNMP v2 is too complex for their liking. This was the reason why it did not become popular.
SNMP v2c is the Community-Based Simple Network Management Protocol version 2. It is defined in RFC 1901 to RFC 1908. Actually, SNMP v1.5 was the initial name given to this protocol. The main difference between SNMP v2 and SNMP v2c is the security model. SNMP v2c uses a simpler community-based security model (found in SNMP v1). Apart from this difference in the used security model, SNMP v2c can be considered almost similar to SNMP v2. In fact, SNMP v2c is now regarded as the de facto SNMP v2.
What is the difference between SNMP v1 and SNMP v2?
SNMP v2 is the successor to SNMP v1. SNMP v2 has different message formats (differences in header and PDU formats) and protocol operations (two extra operations) compared to SNMP v1. SNMP v2 introduced the GetBulkRequest for retrieving a bulk of data at once. Both SNMP v1 and SNMP v2 are now considered obsolete. But, all SNMP implementations still support both SNMP v1 and SNMP v2.
Note that it is expected after an snmpcollector probe restart to not see all the QOS data for some profiles/OIDs/QOS objects, sometimes for hours, and up to 24 hours. This depends on how many devices are being monitored and the polling frequency/number of metrics being collected per instance of the snmpcollector probe.
First, the snmpcollector probe must discover all the devices and components and their related configurations to be monitored.
SNMPv2c allows bulk processing but SNMPv1 does not, so in general, SNMPv2 is preferable in terms of overall monitoring performance. Network factors will play a part as well, e.g., if there is any latency between the snmpcollector probe and the monitored devices so that should be considered and checked.
In some cases, it is recommended to deploy 2 or more instances of the snmpcollector probe, depending on how long it takes to complete the discovery after an snmpcollector probe restart.
Additional Information
SNMP Protocols
SNMPv1 tends to be slower than v2 and v3.
Many important data structures in MIBs are expressed as multi-valued data. One example is the entries in a routing table. With v1, these can only be retrieved by sending multiple GetNextRequest messages. Each message gets a single table item.
snmpcollector Memory and Disk Requirements (broadcom.com)
Other settings
For more information on MAX_BATCH = 5 see Snmpcollector is able to connect and discover the Device but not ALL metrics are being displayed in the OC
Shown below is an example of the page displayed when accessing the snmpcollector H2 DB:
You can run a select count(*) component to get the component count.
Feedback
