Note: Since this article was written, we've changed our pricing model to use virtual user hours rather than node hours. Check out our Pricing guide for more details!
Forecasting and planning your load testing activities can be challenging especially in today’s dynamic project environments. Allowing for adequate time and resources as well as having enough contingency for unexpected testing activities is essential when planning how many node hours you will need using the Flood platform.
What is a node hour and how can they be used?
So, you’re considering load testing in our multi-tenant cloud using Flood’s On-Demand offering. Given we are providing both the load testing platform as well as the infrastructure to generate the load. So, tests with more concurrent users generally cost us more to execute, so our cost to you as a customer is higher.
We have devised a great concept to pass through this cost to you called the node hour. You may have seen in your trial account that you have been granted 5 node hours for your experimentation. But you may be wondering – what is a node hour and how do I use it? That’s a common question that we are happy to answer.
A node hour is the usage one node (server) for one hour, or it’s equivalent. Keep in mind node hours are not fixed to strictly a single node for 1 hour in length. You can actually split this up in a multitude of ways. Some of the many examples include:
- an hour of testing on one node
- 30 minutes of testing on two nodes
- 12 minutes of testing on five nodes
We use the concept of charging in node hours, not number of users like other services, because this makes it very cost-effective and flexible for our customers. By using this model, we charge you for exactly what AWS charges us for: server usage.
Pricing by virtual users is common in the industry, but is limiting in many ways. For one, it can only apply to one test tool and requires several different virtual user thresholds when protocol level and browser level options are offered. Additionally, it is restrictive when needing to run a quick test with a large number of users, for an event like a seasonal promotion or a one-time sale.
Different types of load testing and how they relate to node hours.
There are quite a number of test types that will require different amounts of node hours.
Generally, choosing the correct overall duration for your specific test type will be mandated by your business reporting requirements and overall testing aims. We will cover the main load testing scenarios and what the recommended number of node hours is for each.
- Functional / Unit Tests – you may choose to use one of our supported tools for running some quick tests for Functional or Unit Testing. Since we support Selenium and Flood Element these can be used in small 1 user, 1 node testing scenarios.
- Shakeout Tests – this is the type of scenario used to validate your scripts and target environment are set up properly before running larger, more expensive tests. These are generally shorter tests that last for a period of minutes rather than hours.
- CI/CD Load Tests – your team may be releasing versions of your software in daily builds. In this case running a small load test after a successful build will keep track of overall performance of the software and enable you to pinpoint functionality or fixes that impact performance.
- Full Certification Performance Tests – these types of tests will require full user concurrency load, matching user behaviour and data and often for at least a 1 hour duration.
- Soak Tests – these types of tests will require your standard user concurrency load profile but running for extended periods. For example to answer questions like – How does my app cope with full user concurrency over an 8-12 hour period.
- Stress Tests – this test type usually involves increasing user concurrency over the (Full Certification Performance Tests) test type to see how your application is able to handle more than the expected load and where it falls over.
What key factors impact how many node hours I need?
There are 3 key factors that will impact the amount of node hours that you will need for your testing. They are:
- Tool choice – each of our load test tools have differing levels of user concurrency that can be supported on each load injection node. JMeter & Gatling can support up to 1,000 concurrent users per node which makes tests with high user concurrency more cost effective. On the bottom end of the scale, Selenium can only support up to 5 users per node.
- Test duration – the total duration that a test requires will also impact the number of node hours needed. For instance Soak tests require testing over 8+ hour periods which will increase the number of required node hours substantially – especially if high user concurrency is also needed.
- Number of users – the total number of users will have an impact on node hours too especially coupled with the tool chosen. A 10,000 user test in JMeter or Gatling will only require 10 nodes whereas the same test done with 100,000 users would require 100 nodes.
Functional / Unit Test Scenarios
With the support of Functional tools like Selenium and even our own Flood Element – we can run small 1 node/1 user tests to check basic pass/fail functionality. This may mean setting up a test with the following example characteristics:
Test Duration5 minutesNumber of Users1Number of Nodes1FrequencyOnce per business daySupported Tool UsedAny
This example will launch a Flood via the API which can be setup using a Build Tool such as Jenkins – to run on-demand, at a scheduled time, or after a software build.
The Total Cost for this example would be 0.2 node hours per day, or 4 per month.
Shakeout Test Scenarios
Shakeout tests are usually performed to verify load test script behaviour as well as making sure the target environment is working properly.
You might run multiple shakeout tests especially if a new environment has been built or components have changed. A typical usage scenario would be as follows:
Test Duration12 minutesNumber of Users3Number of Nodes1Frequency3 times per business daySupported Tool UsedAny
These tests are more on-demand in nature and generally initiated manually once the ‘all-clear’ for the environment to be shaken out has been given
The Total Cost for this example would be 0.6 node hours per day, or 12 per month.
CI/CD Load Test Scenarios
Continuous Integration/Delivery testing is usually done whenever a software build is deployed – which may be multiple times a day. These types of tests are generally automated too through a build tool like Jenkins/Travis CI etc. This may mean setting up a test with the following example characteristics:
Test Duration12 minutesNumber of Users20Number of Nodes1Frequency5 times per business daySupported Tool UsedJMeter
Again, this will launch a Flood via the API which can be setup using a Build Tool such as Jenkins – to run on-demand, at a scheduled time, or after a software build.
The Total Cost for this example would be 1 node hour per day, or 20 per month.
Full Certification Performance Test Scenarios
The most common load test scenario is the certification or validation Performance Test. This is the type of test that will require full planned user concurrency volumes.
For example – simulating 20,000 concurrent users for 1 hour with JMeter will have the following usage scenario characteristics:
Test Duration60 minutesNumber of Users20,000Number of Nodes20FrequencyOnce per monthSupported Tool UsedJMeter
The Total Cost for this example would be 20 node hours per month.
Soak Test Scenarios
A soak test is essentially a full validation performance test but run over an extended time frame. This is helpful to see how your system performs over longer periods and can help diagnose potential resource leaks over time.
For example – simulating 20,000 concurrent users for 8 hours with JMeter will have the following usage scenario parameters:
Test Duration8 hours (480 minutes)Number of Users20,000Number of Nodes20FrequencyOnce per monthSupported Tool UsedJMeter
The Total Cost for this example would be 160 node hours per month.
Stress Test Scenarios
Stress tests build on the full validation performance test scenario but will add more concurrent users until a known criteria for the system breaking is observed. This can be that all transactions fail, or that the system returns errors preventing the end user to be able to successfully use the application under test.
For example – simulating 35,000 concurrent users for 1 hour with JMeter will have the following usage scenario parameters:
Test Duration1 hourNumber of Users35,000Number of Nodes35FrequencyOnce per monthSupported Tool UsedJMeter
The Total Cost for this example would be 35 node hours per month.
Adding it all up for yourself.
So in review – we have explained some of the common load testing scenarios and how they would translate to node hour usage on Flood. It’s important for each of our customers to plan out not only the test types they want to run but also the frequency per month. This will enable you to accurately budget your load testing spend accordingly as well as satisfy your monthly testing requirements.
The easiest way to come up with an accurate node hour planning figure – is to note down the types of tests that you require to be run on a monthly basis.Then, you can calculate the node hours for each of these test types, as well as how many of each type you would like to run every month. This will give you the total amount of node hours that you will need so you can confidently select one of our suitable plans.
Keep in mind, you can always roll over unused hours when you renew your plan for the next term (the next month or next year). Additionally, you have the flexibility to upgrade your plan at any time, should you decide that you need more hours.
We have developed a simple way of calculating this with our node hour calculator that does the heavy lifting calculations for you. Feel free to check it out and use it as a basis to select from one of our subscription plans.