Every day, we speak to customers who have created load testing scenarios with popular open source tools like JMeter or Gatling. In fact, almost all customer of Flood use some sort of open source tool for their load test scripting, with JMeter being the most popular of the bunch. As these devoted JMeter users evaluate ways to further enhance what they are doing, they are curious to know "What does Flood provide that I can't get from using JMeter alone?"
No doubt, JMeter is a powerful tool for scripting user scenarios and we are not promoting the need to investigate JMeter alternatives. However, it can fall short in a number of areas that come downstream of running the load test, where a cloud load testing platform like Flood can fill in the gaps. Some of the main areas customers report finding major benefit from Flood are:
Analysis
JMeter offers their own inbuilt functionality around reporting, which many customers are using successfully to create their reports. However, their are a number of limitations to it when using JMeter in an enterprise context:
- Limitations when aggregating the report from a number of JMeter servers, which is required when running tests with high concurrency
- Difficulty seeing historical trends in test results over time
- Requirement to wait until testing is complete to view results (unless building a custom reporting solution like InfluxDB + Grafana)
- Limitations around report customization
- Lack of sophisticated report sharing mechanism (besides export to static file) and ability to control permissions on who can see what reports
Analysis is the end goal of running load tests, without it, we cannot determine what issues we have found and what has caused them. Don't underestimate the importance of strong analysis features to augment JMeter.
Scalability
JMeter does offer its own solution around distributed testing using a Master and Slave set up like the one listed in this diagram:
Typically, users will need to start thinking about a Distributed Load Testing solution when they reach over 1,000 concurrent users in their test, or need to test from multiple different geographic locations. Many users will try their hand at setting up a JMeter Master and Slave configuration before realizing that this process is not going to fit their needs for any number of reasons:
- Too many servers are required, and the maintenance of each server is too burdensome
- Desire to use the cloud and shift to temporary infrastructure to support the tests, and the Master and Slave solution is not ready for the cloud out of the box
- Fear of unnecessary downtime and unplanned outages to the test infrastructure when servers go offline or updates to the software don't go as planned
Once load testing becomes a critical process in your pipeline, you won't want to be stuck with the scenario where everything grinds to a halt because your in house JMeter solution has gone down. Make sure your efforts are focused on building software to support your own business, not reinventing other testing solutions that exist on the market.
Test Management
When using JMeter for one or two tests, test management is not a concern that you likely will grapple with. However, when scaling to running many JMeter tests, several questions will arise as to how you will scale the process including:
- How can you centralize your repository of .jmx test plans you have created so your whole team can easily access them?
- How can you organize tests by projects or product?
- How can you restrict who can see what tests?
- How can you track changes to tests over time?
- How can you integrate to Test Case Management tools like Tricentis's qTest?
If all goes according to plan, you will start to create dozens or even hundreds of tests in JMeter. Don't be caught without the foundation to scale up your load testing initiative.
User Management
JMeter is a tool that is installed locally on one particular machine. Therefore, if someone has access to the installation, they will have access to everything that JMeter has to offer. Thus, it can be tricky to implement the proper controls needed to restrict access to only the assets and functions that a particular user requires. Some questions to ask when it comes to JMeter user management include:
- How will I control who can create, run, edit, and delete my various tests in JMeter?
- How do I ensure 1 team does not constantly use up all of my JMeter resources?
- How can I ensure that my JMeter access is dependent upon my users' status in our SSO provider?
- How can I easily grant access for the dozens, hundreds, or even thousands of users that will need access? And also make sure they have access to all of the relevant projects they will be working without unnecessary effort?
Like Test Management, don't let the best case outcome become your biggest fear: you want to be prepared for scale and onboarding a crowd of interested JMeter users. Additionally, for companies with concerns related to Sarbanes Oxley, there may be a heavy financial cost to not being able to control what users from one department see vs. users from another department.
Integration
JMeter does provide integration to some tools out of the box, including popular CI/CD tools such as Jenkins. However, the functionality in this integration is quite specific, and may not meet the needs of teams implementing their own heavily customized CI/CD Processes. For this, many cloud tools provide a rest API which both allows customization as well as the ability to integrate to additional CI/CD tools outside Jenkins alone.
Other integrations which are often asked for, but lacking (entirely or with regard to advanced functionality) in JMeter include:
- AppDynamics
- New Relic
- Dynatrace
- SSO (SAML, LDAP, etc.)
- Cloud Providers (AWS/Azure)
As you leverage JMeter within your CI/CD pipeline, integrations will essential. You don't want to be caught in a situation where you can't run load tests because your integrations aren't working as expected or can't be customized to your liking.
Support for Other Tools
There is no doubt JMeter is the most popular load testing tool on the market today, open source or otherwise. However, JMeter is not best suited for testing all scenarios, like single pages applications, AJAX calls, or heavy browser side scripting. Additionally, the tool might be too difficult for more non-technical users, and some teams might desire an ability to reuse their functional tests for load tests.
Some of the tools beyond JMeter that we see organizations using to script their load testing scenarios include:
Having a platform which JMeter is integrated to will allow you to offer different teams several options for their testing, all connected to the same centralized platform with consistent reporting. Standardizing on a single load testing solution will have great benefits including reduced overhead and better pricing, but it's likely only possible when multiple scripting options (outside JMeter alone) are available.
Supercharge JMeter with Your Free Trial of Flood
If you're now convinced that your JMeter tests could use the support of a cloud load testing platform, we'd encourage you to sign up for a free trial of Flood. With this trial, you can access our Enterprise package including all of our great features on a limited scale.
If you have any questions about how Flood can help take your JMeter load testing to the next level, please don't hesitate to reach out to us. We are more than happy to share our experiences working with our largest clients leveraging some of the largest JMeter implementations.