Best Practices for IP Infrastructure Monitoring

Products

• CMDB for z/OS
• NetSpy Network Performance
• NetMaster Network Automation
• SOLVE
• NetMaster Network Management for SNA
• NetMaster Network Management for TCP/IP
• NetMaster File Transfer Management
• SOLVE:Operations Automation
• SOLVE:Access Session Management
• SOLVE:FTS

Issue/Introduction

Best Practices for IP Infrastructure Monitoring

Environment

Release:
Component: NMTIP

Resolution

The IP Resource Monitor regularly checks on IP resources, samples and stores their performance data, and sends alerts when they violate performance triggers.

The IP Resource Monitor also provides a single screen from which you can enter diagnostic and performance commands against a resource. Supported resources:

z/OS IP Stacks	OSA	Cisco Channel Cards	2216 routers
Enterprise Extender	VIPA	CSM (Communication Storage Manager)	Address Spaces
(physical infrastructure)   (logical infrastructure)

Step 1 Delete less important resources

When you first set up a NetMaster for TCP/IP region, Express Setup automatically discovers the IP resources that are present and active at the time.

After this initial setup, they are not dynamically updated - you need to update them.

Express Setup is pretty good when finding the big stuff - stacks, OSAs, CIPs and so on. It also finds every address space that was active when it ran. It is highly unlikely you want to monitor all of them - delete all but the most important ones.

As with the IP Node Monitor, don't leave everything that Express Setup finds on the IP Resource Monitor. Don't monitor things just because you can - this wastes space and CPU, and produces unwanted alerts.

To delete an IP resource from the IP Resource Monitor

    Use /IPMON  (M.I) then enter DEL  next to the unwanted resource name 

Unlike IP Nodes, IP Resource Monitor resources of all types are defined individually, not as members of groups.

Stacks and Network Interfaces

Step 2 Add z/OS IP Stacks

To add a stack to the IP Resource Monitor

    Use /IPMON (M.I) then PF4=Add then select a class of  STACK

More information:

See the section "How to Define A Stack" in Chapter 10, "Setting Up Monitoring", in the Administrator Guide .

What's the fuss about?

Seeing a STACK resource as a single line on the IP Resource Monitor does not look too exciting. But behind the scenes, stack monitoring enables many useful things.

• When you monitor a stack, the NetMaster Packet Analyzer can examine all IP connections to/from the stack, producing throughput and other statistics.
  
  
• Stack performance monitoring can include data about:
  
  
  • Connection Workload (from Packet analyzer)
    
    
  • FTP and Telnet Workload (from SMF)
    
    
  • Stack network interfaces (from Packet analyzer and SNMP)
    
    
  • Stack Address Space and Ports (from Packet analyzer)
    
    
  • IP, TCP and UDP activity (from SNMP IP, TCP, UDP MIBS)
    
    
  • Any other stack SNMP MIB attributes (from any IBM MVS MIB)
  The most interesting STACK commands
  
  Not all of the numerous /IPMON Stack commands are equally useful. Try these first.
  
  
• TRS (Traffic Statistics) shows real-time byte/packet throughput grouped by:
  
  
  

  Application Local TCP Port Stack Home Address Stack Interface

  Protocol Protocol by Stack Home Address Protocol by Stack Interface Remote Network

  
  
  This info comes from the Packet analyzer, and can be shown over the last 5 minutes (1-minute intervals), or the last hour (5-minute intervals)
  
  
• PT starts a packet trace, PTV views it, PTI stops it
  
  
• ICF lists IP connections, ITF lists Telnet connections
  
  
• S shows a snapshot of all IP activity, similar to /IPSUM
  
  
• IP, WI, WC, WF, T and H show different kinds of stack performance data.

Step 3 Activate Stack IP, TCP, and UDP SNMP Monitoring

To add IP, TCP, and UDP monitoring to a stack

Use /IPMON (M.I) then UM next to the stack

• Activate/Select IP, TCP, and UDP Monitoring
  
  
• Use F4=Add to add additional attributes
  
  
• Select an attribute, to set optional alerting conditions

Monitor each attribute for at least a few weeks, before setting any alert conditions. This lets history data build up, and gives you an idea of typical hourly values.

Recommended stack IP, TCP, and UDP performance attributes

IP network	TCP protocol	UDP protocol
ipDelivers ipReceives ipDiscards ipReasmRequired ipOutRequests ipOutDiscards ipFragOk	tcpActiveOpens tcpPassiveOpens tcpFailedOpens tcpCurrentConns tcpSegmentsRecvd tcpSegmentsSent tcpResetConnects tcpSegmentsInErr tcpSegmentsRST tcpSegmentsRxmit	udpDgrmsInError udpDgrmsNoPort udpDgrmsReceived udpDgrmsSent

If you are running Enterprise Extender , this uses UDP, so make sure you monitor the UDP attributes.

Step 4 Activate Stack Network Interface Monitoring

To add network interface monitoring to a stack

Use /IPMON (M.I) then UM next to the stack

• Activate/Select Network Interface Monitoring
  
  
• Use F4=Add to add additional attributes
  
  
• Select an attribute, to set optional alerting conditions

Monitor each attribute for at least a few weeks, before setting any alert conditions. This lets history data build up, and gives you an idea of typical hourly values.

Recommended stack network interface attributes

ifInPkts ifInPktsDiscard ifInPktsError ifStatusOper

ifOutPkts ifOutPktsDiscard ifOutPktsError

By default, ifStatusOper will raise an alert if the value is not UP.

OSA (Open Systems Adaptor)

Step 5 Add OSAs

To add an OSA to the IP Resource Monitor

    Use /IPMON (M.I) then PF4=Add then select a class of  OSA 

To update the monitored attributes & alert conditions of an OSA

    Use /IPMON (M.I) then UM then F10=EditLst 
    Use F4=Add  to add an attribute 
    Select an attribute to set alert conditions 

MVS command D M=CHP will display information about OSA channel types, and their status.

More information:

See the section "Define OSA Resources" in Chapter 10, "Setting Up Monitoring", in the Administrator Guide .

OSA Performance Attributes

Most OSA performance attributes are only applicable to specific OSA models, so no specific attribute list will suit all OSAs. However, for your model of OSA, monitor the following aspects:

• TotalOSAErrors
  
  
• DeviceStatus
  
  
• PCI Bus utilization
  
  
• Processor Utilization
  
  
• Read and Write rates
  
  
• Counts of significant error types
  
  
• Packets discarded, sent and received in error
  
  
• Attributes that may affect performance, related to congestion or queue sizes

For complete OSA monitoring, NetMaster for TCP/IP needs to access one of:

1. OSA/SF
   
   
2. The OSA-Express Direct SNMP subagent ( IOBSNMP )

If you don't have OSA/SF or IOBSNMP , you can use the RMF* attributes, which come from RMF.

If you do have OSA/SF or IOBSNMP, use the CHP* attributes from OSA/SF or IOBSNMP, in preference to the RMF* ones.

Cisco CIP host

Step 6 Add Cisco CIP hosts

To add a CIP host to the IP Resource Monitor

    Use /IPMON (M.I) then PF4=Add then select a class of  CIP 

To update the monitored attributes & alert conditions of an CIP host

    Use /IPMON (M.I) then UM then F10=EditLst 
    Use F4=Add  to add an attribute 
    Select an attribute to set alert conditions 

CIP Performance Attributes

Monitor attributes related to:

• Bandwidth utilization
  
  
• Packets received and sent in error
  
  
• Total packets
  
  
• Packets discarded
  
  
• Device availability

More information:

See the sections "Define Cisco Channel Cards" and "Collect Data from Cisco Routers" in Chapter 10, "Setting Up Monitoring", in the Administrator Guide.

Enterprise Extender

Step 7 Add Enterprise Extender

To enable EE monitoring by the IP Resource Monitor

    Use /IPMON (M.I) then PF4=Add then select a class of  EE 

To update the monitored attributes & alert conditions of EE on this system:

    Use /IPMON (M.I) then UM then F10=EditLst 
      Use F4=Add  to add an attribute 
      Select an attribute to set alert conditions 

High, medium, and low priority traffic correspond to VTAM COS (Class of Service) priorities. EE uses a separate UDP port for each priority of traffic.

You should monitor all available EE performance attributes.

EE can only have one definition on a system (because there is only one VTAM).

VIPA

Step 8 Add VIPAs

NetMaster for TCP/IP VIPA resources are used to monitor dynamic VIPAs.

The procedure for defining a VIPA is slightly different to other resources. This is because dynamic VIPAs can move from one stack and system to another.

To add a VIPA to the IP Resource Monitor, use Chapter 10, "Setting Up Monitoring" in the Administrator Guide . See the section, " Define VIPA Resources ".

You should monitor all available VIPA performance attributes.

Performance data alerts

Performance data attributes generally fall into a few main categories.

Category	What is measured?
Availability, Stability, Reliability	Is this resource active, reachable and responsive?
Throughput	How many bytes, packets, datagrams, segments went through, in an interval?
Errors	How many errors occurred in an interval?
Response time	What is the actual response time? What was the lowest, highest, average, in an interval?
Workload	How much work did application end-users do, in an interval?
Capacity, Utilization	How much of its own internal resources was a device using?
Resource Usage	How much of the external system resources was a device using?
Configuration	Device and environmental specific configuration

Alerts can be raised when performance data attribute samples:

• Fall above or below an absolute numeric limit
  
  
• Fall above or below a percentage difference from a moving average
  
  
• Equal or do not equal a particular character value

More information:

Technical document TEC417964, NetMaster for TCP/IP: Performance Data Reference , available from supportconnect.ca.com

Monitoring Maps

You may not want to get performance data alerts at all times. For example, you need to be alerted if a key application has lower than normal throughput during its peak processing hours - but you don't care if this happens during off-peak times.

Monitoring Maps specify the days and hours that a resource will be monitored. Alternatively, you can continue to monitor a resource, but just disable alerting. One Monitoring Map can be attached to many IP resources, so they are all on the same 'schedule'.

More information:

"Monitoring Maps" in Chapter 10, "Setting Up Monitoring" in the Administrator Guide .

See the results

After you do all these steps to define your critical IP network and infrastructure resources, where do you see the results?

The IP Resource Monitor

To access the IP Resource Monitor

    3270        Use /IPMON  (M.I)
    WebCenter   Menu option Monitoring, IP Resources

IP Resource Monitor Display

The IP Node Monitor only includes one type of resource, so it has columns specific to IP Nodes, such PINGRTT.

In contrast, the IP Resource Monitor display includes very different kinds of resources, so its columns have to be more general, about monitoring statuses and so on. It is more like a CA SOLVE:Operations monitor.

Things that determine the status and color of an IP resource on the display include:

• Whether the resource is active
  
  
• Whether the resource can be sampled, or if there are environment errors preventing it being sampled
  (SNMP errors, Packet analyzer not up, attribute is not supported by the device type, like trying to get 10G OSA attributes from older OSAs)
  
  
• Whether or not any of its sampled attributes show abnormal values - you set what you want to be regarded as 'abnormal'.

The colors indicate the status of each IP resource, and are the same as /IPNODE

Status of IP Resource	Color of Line
status OK, no alerts	Green
Unknown or NoAttr error status	White
Timeout, Error or SNMPerror status	Turquoise
outstanding severity 1 alert or serious error during data sampling	Red
outstanding severity 2 alert	Yellow
outstanding severity 3 alert or actual state is INACTIVE	Pink
outstanding severity 4 alert	Blue

While its main general display is not as simple or informative to look at as the IP Node Monitor, the IP Resource Monitor's power is 'hidden' behind it. It is a central control point from where you can enter commands against any resource.

IP Resource Monitor Commands

Type ? next to any /IPMON resource, to see the available commands

• Turquoise commands are specific to the type of resource.
  The available commands depend on the resource type. Some resource types have very few commands - others (like Stacks) have dozens.
  
  
• Green commands are general, common monitor functions.
  
  
• You can run Packet Traces from the IP Resource Monitor.
  SmartTrace commands ( PT, PTV, PTI and PTD ) are supported from the IP Resource Monitor for STACK , ASMON and EE resources.

If the color and/or Alert columns indicate there are current alerts open for this resource, use the AL command to access the Alert Monitor and see them.

The Alert Monitor

If any IP resources have performance-related alerts, these alerts will appear on the Alert Monitor, and be sent to any forwarding destinations.

The Graphical Monitor

The Graphical Monitor (3270-only) is fed by the IP Resource Monitor, and can be a more suitable way to display resource statuses than the IP Resource Monitor.

See The CA NetMaster for TCP/IP Handbook to learn about exploring the Graphical Monitor.