NetMaster SSI abends for PACKET ANALYSER DATABASE limit

Products

CMDB for z/OS NetSpy Network Performance NetMaster Network Automation NetMaster Network Management for SNA NetMaster Network Management for TCP/IP NetMaster File Transfer Management SOLVE

Issue/Introduction

Started to see *NF5Z01 DB/DS: DYNAMIC/DDB STORAGE IS AT 80% repetitively until
NF7Y10 PA DATABASE/DATASPACE DYNAMIC/DDB AT LIMIT, AUTO SHUTDOWN PROC

Set all expiry periods to '0' under /IPAPPLS but even then NMIP keeps cycling through NF5Z01 and NF5Z02 messages until it hits NF7Y10 PA DATABASE/DATASPACE DYNAMIC/DDB AT LIMIT, AUTO SHUTDOWN PROC again.

From the packet analyser DDB usage command that TCX (then KMU and KMS) types are what was driving up storage - just not sure how to diagnose what was driving those.

Also - is there a way to resize the PA Database as we "occasionally" hit that 80% on this LPAR every few months...

04.17.00+ssis sh paddbstg level=4
04.17.00 NF8O10 PACKET ANALYSER DYNAMIC DATABASE USAGE INFORMATION FOLLOWS...
04.17.00 NF8O11 SIZE: 512 MB, (536870912 BYTES), 131072 PAGES
04.17.00 NF8O12 ALLOC: TOTAL: 131071 USED: 120946 (92.28%) FREE: 10125 (7.72%) HWM: 126295 (96.35%)
04.17.00 NF8O13 REQS: ALLOC: 2951942 (FAILED: 0) FREE: 2831043
04.17.00 NF8O20 DDB USAGE INFORMATION FOR STACK: TCPIP FOLLOWS...
04.17.00 NF8O21 USAGE: PAGES: 120946 (92.27% OF TOTAL) (100.00% OF USED)
04.17.00 NF8O22 PAGE USAGE STATISTICS...
04.17.00 NF8O23 TYPE COUNT TYPE COUNT TYPE COUNT TYPE COUNT
04.17.00 NF8O24 STK 1 IF@ 10 R@N 79 IPR 2
04.17.00 NF8O24 MSC 48 ICM 0 UDP 0 TCP 0
04.17.00 NF8O24 XAD 1 SDC 26 TCX 109774 TCY 229
04.17.00 NF8O24 KMU 1998 KMS 8691 THT 17 UCX 21
04.17.00 NF8O24 UCY 3 UHT 2 EEI 14 UNK 0
04.17.00 NF8O24 HTB 30 *MPB 13
04.17.00 NF8O24 NVU 0 FRE 0 ALC 0 BEF 0
04.17.00 NF8O25 RECORD TYPE STATISTICS...
04.17.00 NF8O26 TYPE COUNT TYPE COUNT TYPE COUNT TYPE COUNT
04.17.00 NF8O27 AIN 21 EEI 7 FTPT 7 IFA 12
04.17.00 NF8O27 IFC 14 IHA 14 IHX 20 IPR 9
04.17.00 NF8O27 IPX 16 RAD 143 RNW 15 SDC 552
04.17.00 NF8O27 STK 1 TOW 36 TPT 76 UPT 5
04.17.00 NF8O27 XAD 7 EEI 7 SDX 3 SDT 0
04.17.00 NF8O27 TCN 218931 -A 706 -H 218225
04.17.00 NF8O27 UCN 21 -A 17 -H 4
04.17.00 NF8O27 ISA 0 FISA 0 TSA 0 HSA 0
04.17.00 NF8O28 HASH TABLE STATISTICS...
04.17.00 NF8O29 TYPE #-RECS #-AP #-E #-U MIN-LEN MAX-LEN AVG-LEN
04.17.00 NF8O30 AIN 21 61 41 20 1 2 1.05
04.17.00 NF8O30 RAD 143 3049 2908 141 1 2 1.01
04.17.00 NF8O30 RNW 15 101 86 15 1 1 1.00
04.17.00 NF8O30 SDC 552 16381 15840 541 1 3 1.02
04.17.00 NF8O30 EEI 7 997 990 7 1 1 1.00
04.17.00 NF8O30 SDX 3 61 59 2 1 2 1.50
04.17.00 NF8O30 SDT 0 61 61 0 0 0 0.00
04.17.00 NF8O30 TCN 218931 16381 0 16381 5 18 13.36
04.17.00 NF8O30 TOW 36 251 218 33 1 2 1.09
04.17.00 NF8O30 TPT 76 5101 5025 76 1 1 1.00
04.17.00 NF8O30 UCN 21 2029 2008 21 1 1 1.00
04.17.00 NF8O30 UPT 5 2029 2024 5 1 1 1.00
04.17.00 NF8O30 XAD 7 199 193 6 1 2 1.16
04.17.00 NF8O30 ISA 0 61 61 0 0 0 0.00
04.17.00 NF8O31 TCN STATE STATISTICS...
04.17.00 NF8O32 STATE #-RECS %
04.17.00 NF8O33 UNKNOWN 222 0.10
04.17.00 NF8O33 LISTEN 0 0.00
04.17.00 NF8O33 SYN SENT 3 0.00
04.17.00 NF8O33 SYN RCVD 0 0.00
04.17.00 NF8O33 ESTAB 481 0.21
04.17.00 NF8O33 FIN WAIT 1 0 0.00
04.17.00 NF8O33 FIN WAIT 2 0 0.00
04.17.00 NF8O33 CLOSE WAIT 0 0.00
04.17.00 NF8O33 CLOSING 0 0.00
04.17.00 NF8O33 LAST ACK 0 0.00
04.17.00 NF8O33 TIMED WAIT 0 0.00
04.17.00 NF8O33 CLOSED 1 0.00
04.17.00 NF8O33 CONN FAIL 218224 99.67
04.17.00 NF8O33 BAD STATE 0 0.00
04.17.00 N11907 *END*
04.18.00 NS2101 NMSSI SHUTDOWN IN PROGRESS. SSID: NMSSI
04.18.01 N98102 SSI CONNECTION STOPPED TO SSID=NMSSI REASON: SSI REGION SHUTDOWN

then once it cleared up

04.42.48 NF8O10 PACKET ANALYSER DYNAMIC DATABASE USAGE INFORMATION FOLLOWS...
04.42.48 NF8O11 SIZE: 512 MB, (536870912 BYTES), 131072 PAGES
04.42.48 NF8O12 ALLOC: TOTAL: 131071 USED: 749 (0.58%) FREE: 130322 (99.42%) HWM: 751 (0.57%)
04.42.48 NF8O13 REQS: ALLOC: 721 (FAILED: 0) FREE: 19
04.42.48 NF8O20 DDB USAGE INFORMATION FOR STACK: TCPIP FOLLOWS...
04.42.48 NF8O21 USAGE: PAGES: 749 (0.57% OF TOTAL) (100.00% OF USED)
04.42.48 NF8O22 PAGE USAGE STATISTICS...
04.42.48 NF8O23 TYPE COUNT TYPE COUNT TYPE COUNT TYPE COUNT
04.42.48 NF8O24 STK 1 IF@ 9 R@N 70 IPR 2
04.42.48 NF8O24 MSC 43 ICM 0 UDP 0 TCP 0
04.42.48 NF8O24 XAD 1 SDC 19 TCX 400 TCY 3
04.42.48 NF8O24 KMU 34 KMS 85 THT 17 UCX 16
04.42.48 NF8O24 UCY 3 UHT 2 EEI 14 UNK 0
04.42.48 NF8O24 HTB 30 *MPB 13
04.42.48 NF8O24 NVU 0 FRE 0 ALC 0 BEF 0
04.42.48 NF8O25 RECORD TYPE STATISTICS...
04.42.48 NF8O26 TYPE COUNT TYPE COUNT TYPE COUNT TYPE COUNT
04.42.48 NF8O27 AIN 19 EEI 7 FTPT 0 IFA 9
04.42.48 NF8O27 IFC 14 IHA 14 IHX 14 IPR 9
04.42.48 NF8O27 IPX 13 RAD 130 RNW 11 SDC 445
04.42.48 NF8O27 STK 1 TOW 25 TPT 81 UPT 3
04.42.48 NF8O27 XAD 7 EEI 7 SDX 3 SDT 0
04.42.48 NF8O27 TCN 796 -A 796 -H 0
04.42.48 NF8O27 UCN 16 -A 12 -H 4
04.42.48 NF8O27 ISA 0 FISA 0 TSA 0 HSA 0
04.42.48 NF8O28 HASH TABLE STATISTICS...
04.42.48 NF8O29 TYPE #-RECS #-AP #-E #-U MIN-LEN MAX-LEN AVG-LEN
04.42.48 NF8O30 AIN 19 61 43 18 1 2 1.05
04.42.48 NF8O30 RAD 130 3049 2920 129 1 2 1.00
04.42.48 NF8O30 RNW 11 101 90 11 1 1 1.00
04.42.48 NF8O30 SDC 445 16381 15941 440 1 2 1.01
04.42.48 NF8O30 EEI 7 997 990 7 1 1 1.00
04.42.48 NF8O30 SDX 3 61 59 2 1 2 1.50
04.42.48 NF8O30 SDT 0 61 61 0 0 0 0.00
04.42.48 NF8O30 TCN 796 16381 15604 777 1 3 1.02
04.42.48 NF8O30 TOW 25 251 227 24 1 2 1.04
04.42.48 NF8O30 TPT 81 5101 5020 81 1 1 1.00
04.42.48 NF8O30 UCN 16 2029 2013 16 1 1 1.00
04.42.48 NF8O30 UPT 3 2029 2026 3 1 1 1.00
04.42.48 NF8O30 XAD 7 199 193 6 1 2 1.16
04.42.48 NF8O30 ISA 0 61 61 0 0 0 0.00
04.42.48 NF8O31 TCN STATE STATISTICS...
04.42.48 NF8O32 STATE #-RECS %
04.42.48 NF8O33 UNKNOWN 429 53.89
04.42.48 NF8O33 LISTEN 0 0.00
04.42.48 NF8O33 SYN SENT 0 0.00
04.42.48 NF8O33 SYN RCVD 0 0.00
04.42.48 NF8O33 ESTAB 362 45.47
04.42.48 NF8O33 FIN WAIT 1 5 0.62
04.42.48 NF8O33 FIN WAIT 2 0 0.00
04.42.48 NF8O33 CLOSE WAIT 0 0.00
04.42.48 NF8O33 CLOSING 0 0.00
04.42.48 NF8O33 LAST ACK 0 0.00
04.42.48 NF8O33 TIMED WAIT 0 0.00
04.42.48 NF8O33 CLOSED 0 0.00
04.42.48 NF8O33 CONN FAIL 0 0.00
04.42.48 NF8O33 BAD STATE 0 0.00
04.42.48 N11907 *END*

Environment

Release : 12.2

Component : CA NetMaster Network Management for TCP/IP

Resolution

The reason for the database filling up can be attributed to one specific thing, and that is due to having 218225 connection HISTORY records.
As seen, the TCN is the TCP Connection Record where there were a TOTAL of 218931, of which only 706 were ACTIVE (as seenin the -A). -H indicates History records and as seen that is 218225.

04.17.00 NF8O27 TCN 218931 -A 706 -H 218225

The solution to this is to eliminate non-essential history records by setting '0' in IPAPPLS.

As to question on making the packet analyzer database larger, that can be accomplished by setting this command within the SSI parms:

PDYNDBSIZE 1024

As seen in the latest $$SYSPRO results (by the way, SSI statistics are always best to issue SSIS SH PADDBSTG LEVEL=4) the record counts are very minimal.

Record Type Count
ADF 14
AIN 41
EDF 0
EEI 7
EGN 22
EIG 0
EIN 0
ETN 39
HCN 197
IFA 59
IFC 14
IHA 14
IHX 39
IPR 12
IPX 18
MSR 2
RAD 624
RNW 40
SDC 1375
STK 1
TCN 1068
TDF 40
TIN 12
TOW 620
TPT 174
UCN 28
UPT 369
XAD 7

The results of the SSIS SH PADDBSTG command is primarily used for support/development purposes, as such, none of those fields are documented externally.
In 99.99% of cases where the SSI PA Storage / data space became too large was caused by one of the following record types, usually in combination of each together:

RAD - Remote Network addresses. Very rare though that you see 100K+ of these

SDC - Sysplex distributor record, these can accumulate but again very rare if there's a large number here

TPT - TCP Ports inuse (server/client), we have mechanisms to clean these up thus very rare if there's large numbers of these

TCN - TCP Connection record where all connections are kept. 98.99% of the time this is the problem record as customers keep historical records and thus TCN's (open and closed) keep building and building to where anything over 250K becomes problematic. That was the situation here.