Web Services tuning for error "Requested 250keys but received 50keys only"

book

Article ID: 48800

calendar_today

Updated On:

Products

CA IT Asset Manager CA Software Asset Manager (CA SAM) ASSET PORTFOLIO MGMT- SERVER SUPPORT AUTOMATION- SERVER CA Service Desk Manager - Unified Self Service KNOWLEDGE TOOLS CA Service Management - Asset Portfolio Management CA Service Management - Service Desk Manager

Issue/Introduction

Description:

If using Web Services, you may sometimes see messages such as the following in the SDM stdlogs:


04/29 11:29:19.64 MY_SERVER1 bpvirtdb_srvr 9988 ERROR vdbmisc.c 1663 Requested 0keys but received 50keys only04/29 11:29:19.67 MY_SERVER1 bpvirtdb_srvr 9988 ERROR vdbmisc.c 1663 Requested 100keys but received 50keys only04/29 11:29:19.67 MY_SERVER1 bpvirtdb_srvr 9988 ERROR vdbmisc.c 1663 Requested 132keys but received 50keys only

This message indicates that Web Services is making requests that the domsrvr is currently unable to fulfil, typically because it is overloaded.

This message may usually be considered as a "notification" only. Especially if seen infrequently. It is not a fault and will correct itself.

However, if you are a heavy users of Web Services and see this message frequently, it may indicate that the environment is under-resourced. If the situation continues and the above message floods the system, it may result in the Web Services refusing to accept any more requests. This messagemay then be seen:


04/29 11:38:23.19 MY_SERVER1 boplgin 7572 ERROR bpremref.c 1015 Version logon to process agent_intf#4204	|BP_LOGON 
unsuccessful after 100 attempt
s - discarding 1 queued messages

In this specific case, we know, because of the keys error, that Web Services has overloaded the domsrvr process. But note that the above message alone is a general message typically indicating process overload and may have other causes.

You will typically see a large number of Web Services requests in the stdlog at the same time, in the form of:


04/26 13:20:28.62 MY_SERVER1 sda 4204 SIGNIFICANT sda.cpp 2054 Web Services session created 103325347104/29 13:20:35.77 MY_SERVER1 sda 4204 SIGNIFICANT sda.cpp 2054 Web Services session ended 1033253471

GENUINE LOAD EXAMPLE
In a case from a user site where the known cause was a Web Services code loop, the actual numbers that led to the above BP_LOGON error were:

DETAILS FROM STDLOG
From 04/26 12:48:13.16
to 04/26 12:49:11.34
There were 18,987 Web Services entries in the forms of:
Web Services session created - 6,769
Web Services session ended - 12,218
Total time 6 minutes 22 seconds or 382/minute or 49.7/second.

where the Web Services was using the default primary server domsrvr.

You may also see other symptoms of high system use:


04/29 12:04:28.25 MY_SERVER1 slump_nxd 9316 SIGNIFICANT server.c 3767 SLUMP Build(SDr12_6-165): Up Time(4 day 19 hr 52 min 15 sec)	 
Sent (4414519) Received (4415811)
Ports in Use (85) Processes (82) Fast Channel (38)

Note that the failure was not immediate. It took several minutes of flood activity, on top of regular domsrvr activity, before failures at this volume occurred. Also note that even though the root cause here was a message flood, Web Services is capable of handling this load if additional architecture is configured.

Solution:

Identify the cause of the message. Consider these 3 possibilities:

  1. Web Services flood.
    Invalid Web Services activity, usually caused by a coding error, is generating volume far in excess of what was scoped for.
    Solution: Prevent the flood.


  2. Dedicated Web Services domsrvr.
    Solution: The volume may be valid, in which case dedicate a domsrvr solely to Web Services via the Options Manager "webservice_domsrvr" option.
    Preferably on a secondary server. See TEC581936 How to Add and Configure a "Web Services Domsrvr."


  3. Additional Web Services servers.
    If symptoms occur on a really heavy-use site, then configure additional Web Services servers with multiple domsrvrs each. Web Services splitting out of load will need to be done, as the option in (2) only allows pointing to a single domsrvr. Two ways among others are either at the Web Services code level specifying the domsrvr to connect to; or hardware redirect. It is strongly recommended that you engage CAT Services or a CAT Business Partner if working with these volumes.

    Or if the message is infrequent and the site scoping is deemed to be adequate - then no action required.

Environment

Release: UAPMAC990JPP-12.6-Asset Portfolio Management-Asset Configuration
Component: