Spectrum 10.4.3 terminated with "ERROR TRACE at CsIHPrtIPLS.cc(1090): Waited 60000ms for IPLS evaluate lock ..." after REST query

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

Following a REST read query, Spectrum was terminated with the following error:

Nov 23 18:21:27 "ERROR TRACE at CsIHPrtIPLS.cc(1090): Waited 60000ms for IPLS evaluate lock for mh: 0x93aff9, continuing without lock

Nov 23 18:21:27 ERROR TRACE at CsIHPrtIPLS.cc(1090): Waited 60000ms for IPLS evaluate lock for mh: 0x93b03b, continuing without lock

An application fault has occurred.

Saved full memory diagnostic file to 'E:\CA\Spectrum\SS\support\SpectroSERVER.mdmp'

Saved compact diagnostic file to 'E:\CA\Spectrum\SS\support\SpectroSERVER_20211124_1519.dmp'

Appending thread state information to .moot.trace file

Exiting.

Environment

DX NetOps Spectrum 10.4.3

Cause

This might be due to a REST call being made (GET-over-POST) with Failure message:

Error.NoAvailableDomains was given for the REST call (0x92a552 is a device model handle).

Resolution

Analysis of the dumps shows the crash was caused by an invalid REST query where an incorrect attribute data type mismatch occurred while setting an attribute to the DB.

The crash is a REST read request where the SS is trying to create a huge memory chunk (and assuming the wrong length because of the mismatched attr type). This is causing an out of memory crash.

We are adding further validation checks on attribute inputs to avoid these type of crashes so this requires additional coding which will be targeted for the next release.

However, the interim solution is to check the data inputs to the REST queries used to write attributes and ensure they're valid for the query.