The Flood platform prides itself on being tool agnostic and supports a number of leading open-source tools as well as it's own simple Test Builder tool.
Jmeter and Gatling are purpose built performance tools which generate load on the HTTP protocol level and then we have Selenium - which works differently in the way that it generates load on the Browser UI level. Both types of tools are very useful for performance testing but each one (like anything in life) have trade offs. That trade-off is complexity vs concurrency.
A lot of today's web applications are significantly complex - try visiting amazon.com with the Chrome Developer Tools add-in enabled on the network tab. You can easily view every request that the browser makes on the front page for images, dynamic content (XHR), javascript libraries etc. and as of the time of writing - contains 349 individual requests.
If we were using Jmeter or Gatling to simulate simply going to the front page - we'd have to script every single request and worry about correlating session ID's etc. There's more bad news than good news with this - the bad news is:
- It will take even the most expert scripting wizard a significant amount of time to script this.
- If anything changes i.e. code is written and functionality on the page changes - guess what? You'll need to update your script!
The good news:
- You will be able to simulate a lot more concurrent users scripting this front page on the protocol level.
In Selenium, you would focus on interacting with the page objects themselves and not worry about the individual requests - which makes it a significantly easier scripting task. The bad news is with this method - concurrency suffers due to the amount of resources a single browser level user needs to have available.
At the present time - a single Flood node can only support 5 concurrent Selenium users as opposed to approximately 1,000 Jmeter or Gatling users per node - A significant difference.
Now, different technologies aside - let's take a look at each tool's advantages and disadvantages in terms of fit for purpose.
Flood - Test Builder
The Test Builder built into the Flood user interface - is a great tool for very simple HTTP based load tests. It's aim is to get up and running very quickly for new users without needing the knowledge to script a test in Jmeter, Gatling or Selenium.
PROs
- You can very easily add the URLs you would like to test.
- Easily configure the load test parameters such as number of users, AWS region, ramp-up time and test scenario duration.
- Run a simple test within minutes.
- Can build simple tests and then export to Jmeter for further advanced scripting techniques
CONs
- Not suited for advanced scenarios such as using parameterization or using dynamic data sets.
- No support for Browser Level Users
Apache Jmeter
Jmeter is arguably the most feature-rich and powerful open-source load testing tool available at the moment. It allows you to build upon basic tests created with our very own Test Builder with more advanced techniques such as:
- Full paramaterization of variables
- Capturing and verifying a wide variety of responses for manipulation & re-use
- Enables the development of quite complex scenarios
- And many more features
PROs
- Open-source.
- Platform independent (Java based)
- Proven in Enterprise-level testing
- Support for a wide variety of standard and 3rd party add-ins which enables you to cover almost all possible types of web performance testing scenarios.
- Ability to create very complex load tests (if needed).
- Ability to use a Ruby DSL code rather than the default tree view. This is accomplished by using the Ruby-jmeter gem which also allows you to run tests in Flood.
CONs
- No support for Browser Level Users
- Does not run client side Javascript
Gatling
Gatling is a well-proven and powerful open-source load testing tool. Using the intuitive Scala scripting language - it helps even first-time users get up and running fairly quickly.
It allows you to build upon basic tests created with our very own Test Builder with more advanced techniques such as:
- Full paramaterization of variables
- Capturing and verifying a wide variety of responses for manipulation & re-use
- Enables the development of complex scenarios
- And many more features
PROs
- Open-source.
- Easy to use and descriptive Scala scripting language used to create tests
- Proven in Enterprise-level testing
- Ability to create very complex load tests (if needed).
CONs
- No support for Browser Level Users
- Does not run client side Javascript
Selenium
Selenium is a very powerful open-source testing tool mainly used for automated functional testing via interacting with browser level objects. It can be very useful even for load testing as it allows users to re-use existing functional tests and run them with virtual concurrent users.
One of the main attractions in choosing Selenium over a protocol level load testing tool such as Jmeter or Gatling - is the ability to create your scripts in a user based functional flows rather than looking at how to simulate users on a protocol level. Protocol Level User's are often much harder for complex applications.
PROs
- Open-source.
- Supports a number of languages to develop scripts; including Java which is supported by Flood.
- Ability to create tests using a 'functional flows' instead of diving into the protocol and coding every single request.
- Ability for virtual users to execute client-side javascript.
CONs
- As threads are GUI/Browser level - which uses up more resources - concurrency is impacted to 5-10 users per Flood load injection node.