Taurus PassFail criteria for each test case does not work in BlazeMeter

Products

• BlazeMeter

Issue/Introduction

We are currently working on implementing CI/CD for one of our projects. We are using framework which uses Taurus CLI to trigger the BlazeMeter test and using Jenkins performance plugin to collect and display the stats using stats.xml file. As part of requirement, we have to set the Pass/Fail Criteria for each request/sample available in the scenario. In Taurus.yml file, we have provided the Pass/Fail criteria as below.

provisioning: cloud
reporting:
  - module: passfail
    criteria: - p99 of API_AddSubmission>10s, continue as failed
  - module: final-stats
dump-xml: stats.xml

In the above, "API_AddSubmission" is a request label. We are noticing that the build is not getting failed if the p99 of the above request crosses the thresold value. Hence, we are not able to set the passfail criteria for each test case. Could anyone please help how to set up passfail criteria for each test case/request in the taurus.yml file? 

Environment

Release :

Component : PACKAGE CA BLAZEMETER

Resolution

In researching this issue, I found out by our R&D team that using the Pass/Fail Criteria from the Taurus Script is not supported in BlazeMeter. This is a known issue that needs to be addressed by Taurus. We have an open request with the Taurus development team, but there is no commitment from the team on when this will be resolved. For now, you must use the Failure Criteria feature in BlazeMeter.