JMeter is the most popular open source load testing tool on the planet, and one of the main reasons for its widespread adoption is its flexibility. A key to any good JMeter load test is the ability to assert that the response from the application fits any number of criteria the tester specifies.
To achieve this aim, JMeter comes out of the box with a variety of assertions at the disposal of the tester. As each assertion is built to achieve a different outcome, it is an important step in each JMeter user’s evolution to understand the main types of assertions, including when and how to use them.
In this JMeter Tutorial, you will learn when and how to use the following common JMeter assertion types to improve the precision of your JMeter load tests:
- Response Assertion
- JSR223 Assertion
- Duration Assertion
- Size Assertion
- Compare Assertion
- File Type Assertions (JSON Assertion, HTML Assertion, MD5Hex Assertion, XML Assertion)
What is a JMeter Assertion?
JMeter assertions are the component of a test that allow a user to validate that the response JMeter receives matches expected criteria. Much like Selenium assertions or even manual testing assertions (i.e. expected results), they make sure the user is aware of any inconsistent or unexpected responses from the target application.
JMeter assertions are a powerful tool, but their effectiveness is contingent on both the assertion criteria selected as well as the accuracy of the preceding request. . The assertion validates that the response from the application is received as expected, but that response typically relies on the correct formulation of a previous request.
What type of JMeter assertion should I use?
There are a variety of JMeter assertions available. In this tutorial we have focused on the 6 most common JMeter assertion types to highlight when to use these powerful test components.
Response Assertion
The Response Assertion is one of the most commonly used assertions in the toolkit. It can look at the response from the application and ensure it matches simple patterns such as containing text, response codes, or response headers.
With this flexible assertion, it’s easy to quickly validate if a service is returning the text desired (such as an order confirmation after the checkout process). Additionally, you can use the “Not” operator to make negative checks such as making sure the response code from the application is not equal to 400, 503, 504, etc.
JSR223 Assertion
More seasoned JMeter users will appreciate the flexibility of the JSR223 assertion for their complex testing needs. Much like the JSR223 timer, this assertion requires knowledge of a scripting language like Groovy to leverage the power of this component. With the proper script running, you can perform complex transformations and parse responses to validate that responses match the patterns desired.
Duration Assertion
With a simple duration assertion, testers can ensure that responses are fully received in the amount of time expected. If the response takes longer than the set duration in milliseconds, then the assertion will fail. This will also mark the transaction as failed in the Flood dashboard.
This tool can be useful when the business requires a specific response time threshold to be met for each step. The duration assertion can be a useful tool to pair with other assertions.
Size Assertion
The size assertion is another simple assertion that allows you to set limits for the size of the response. . Responses that are significantly too large or too small can be set to cause transaction failures to make sure that any response is within the acceptable bounds of file size.
Compare Assertion
The compare assertion allows a user to determine if the response from an application matches with the expected result by using a regular expression (regex) to logically assess the response. Like other assertion types, it is possible to make this a positive or negative assertion (i.e. verify that the response does or does NOT meet certain expectations).
Some users may prefer to use the JSR223 assertion for its added flexibility; however, the compare assertion allows those who don’t know how to code (or don’t wish to) an easier and faster option to learn.
File Type Assertions
File type assertions are another basic form of assertion that allows a user to check that the response is a valid file of the type indicated. The file types with their own specific assertions include: JSON Assertion, HTML Assertion, MD5Hex Assertion, and XML Assertion.
What to do when a JMeter assertion Fails
There are various schools of thought about how to treat an assertion. Some testers will stop a test entirely when an assertion fails. Others will say you should stop that thread or loop. Others will recommend that you continue while noting the error for reports. Each approach has its own benefits depending on the purpose of the test.
Note that these settings are controlled at the Thread Group level and would relate to all samples and subsamples:
When to use multiple JMeter assertions
It is possible and very common to use multiple assertions with each JMeter sampler to create a more powerful combination than any component can provide on its own. Since the assertions are executed in series, the failure of any one assertion can “trip the circuit” and cause the selected Thread Group behavior to trigger (stop the thread, stop the test, etc. depending on the configuration).
For example, asserting that a file matches a valid type can be a wise step before running other steps to process or extract a response. If the response is invalid, the script can skip to the next sample or stop entirely, depending on the thread group configuration.
What scope should be used for JMeter assertions?
Many of the most popular assertions listed in this tutorial (including size, duration, and response assertions) can be applied at various levels of scope.
Adding an assertion as a child of the Test Plan will cause the assertion to be applied after every sample in that test plan. To apply the assertion to one sample only, add it as a child of the sample. Note that assertions are always processed after the sample is executed.
Even when the assertion is a child of a sample, there can be variation as to its scope. When applied to a main sample only, it will not affect sub-samples. Similarly, if applied to sub-samples only, it will not affect main samples. Selecting the option for “main sample and sub-samples” will ensure an assertion is applied to the entire scope of that sample.
How are assertions reported in my test?
Once set, assertions will pass or fail the transaction you’ve set them against. If an assertion fails, the whole transaction is marked as failed and you’ll be able to see it as an error.
In JMeter, you’ll be able to see this by looking through your results using a listener. Here’s what it looks like in the View Results Tree listener:
In Flood, you’ll also be able to see what happens with the assertions.
Here’s a sample flood that we ran with both passing and failing assertions. The transaction “01_HomePage” has a response assertion that passes, and the transaction “02_Step1_ClickStart” has a response assertion that causes the transaction to fail.
When clicking on the arrow to the right of the failing transaction, you’ll be able to see the exact request headers as well as the response sent back that was causing the error.
How to Use the Dummy Sampler
Making sure your assertions are set up properly can be a tricky and time consuming process when using the real responses from your application. With the JMeter dummy sampler, you can instead simulate a response from your application and use that as the input to test whether your assertion is properly configured.
To give you an idea of how the dummy sampler can be used, we have set up a repository with several sample JMeter scripts leveraging the dummy sampler. In each example, you can see how the dummy sampler can be used to quickly verify that your assertion acts as anticipated. For example, we have shown how you can simulate some different types of responses so you can quickly build some assertions into your test.
JMeter assertions: A powerful tool in the arsenal of any load tester
Assertions are critical to writing a good load test script that provides fast feedback on application performance. When assertions are not properly implemented, tests give inconsistent or nonexistent feedback on how the application is performing. Fast feedback alerts you when a regression has occurred so that you can start the process of determining the root cause.
The best way to become familiar with assertions is to see examples in action and practice on your own using the dummy sampler (as shown in this post). Use our sample scripts to get familiar with all of the different assertions JMeter has to offer, and don’t be too shy to start using some of these assertions in your tests today!