An Introduction to Load Testing

For the Tricentis User Conference 2020, Flood co-founder Tim Koopmans and I teamed up to discuss our thoughts on how someone can get started with load testing from scratch. Watch the presentation recording below, or keep reading for a look at some of the topics we discussed.

‍

"The right background" for load testing

‍

Tim: I’ve been doing load testing for many years now, I would say since 2004 at least, and I’ve really enjoyed that whole time being able to get in and figure out all things related tosoftware and application performance.

Now I wasn’t always a performance tester. I grew up in and currently live in a small country town in Australia, and after joining the military straight from school, and of course doing lots manual tasks in that 12-year career, it was only natural that I developed a love of automating things. I never figured out how to automate cleaning, or kitchen duty, but there you go.

From my early days of using tools like perl to scrape websites and trying to win bids with early versions of ebay, through to an 8-year consulting career doing performance testing and engineering with both commercial and open source tools, all the way through to founding and now running Tricentis Flood – performance as a topic is near and dear to my heart.

‍

Nicole: I was born in Metro Manila in the Philippines (which has 14,000 more people per km2 than Mumbai does). Like many of you, I learned English as a foreign language, in school. I have a degree in Economics, and when I discovered load testing I had a job in a completely different field– I was working for an airline and making sure engine components were properly maintained and repaired. I’d always loved gadgets and technology, so it didn’t take long before I decided to become a load tester.

Today, I have over 8 years of experience in software testing, and most of it was spent consulting for companies in Australia and the Netherlands, where I live now, helping to make their applications faster. I now work at Flood as a performance testing advocate. I still like helping people run their load tests, but I also like educating people about performance testing.

As you can see, Tim and I came from very different backgrounds and different parts of the world. But somehow, we still both ended up becoming load testers at Flood.

‍

Why should you become a load tester?

You get to help out with cool projects

Tim:

You may have heard of Flood. But even if you haven’t, we’re sure you’ve heard of one of our favourite customers. In the last few years, we have been helping Hotstar, one of the largest streaming services in the world, load test their systems in preparation for the Indian Premier League.

Hotstar uses an open-source tool that we support, Gatling, to create scripts to simulate users tuning in to watch cricket games. Then, they use Flood to create a “Game Day” simulation. They scale up those load tests and run 5 million users on thousands of machines in the cloud. We help them coordinate execution, spot problems quickly with real-time results from all those machines while the test is still running, and identify bottlenecks so that they don’t occur in production, when India’s playing.

But other than being able to help out with cool projects like Hotstar, why would you want to be a load tester?

Performance pays

Nicole:

Here are 261,768 more reasons why. The average performance engineer’s salary is 71% more than that of an average software tester in India, according to Payscale.com. We all know that money isn’t everything, but this is a clear signal from the market about the importance of performance-related skills. Adding load testing to your CV as a tester makes you more attractive to employers. It’s a great way to differentiate yourself in an over-saturated market.

Everybody fails

Tim:

Salary aside, performance is an aspect of software that is often overlooked, but it’s only becoming more important in a world where everything is going online. People are being born who have never known what it’s like not to have the internet, and consumers in general expect applications to be responsive and stable. Would-be customers who can’t get an e-commerce site to load within 3 seconds may switch to a competitor’s site, and we all know there are many of them. Performance-related outages can be seriously damaging to a company’s reputation, and social networking amplifies customer dissatisfaction by effectively giving everyone a megaphone to complain into.

Software functionality is still important. However, if an online store goes live with no critical functional defects, but it fails when the 100th person logs into it, or it takes 10 seconds to load the homepage, can you really say that it was ready for production?

Load testing is a crucial part of ensuring software quality, and quality is the responsibility of everyone on a team, even if the word “performance” isn’t in your job title.

Remote working is a possibility

Nicole:

Then there are the side benefits of being in load testing. The photos on the right are real ones from the Flood team. That’s me on the upper left, exploring a castle in Belgrade, Serbia, and that’s Tim in the bottom right, taking a photo in San Francisco, USA. We’re a fully distributed team on Flood, which means we can work from anywhere. Flood is a web app, so you don’t need to install anything– I’ve been known to start load tests from my iPad in a café. Load testing on the cloud frees you from being tied to running tests from the office. You can start load generators when you need them and shut them down when you don’t– all from a web browser. It’s a job that can come with a great work/life balance, if you find the right company.

Load testing can also be a great way to move to another country. It’s what’s allowed me to move to other countries– twice. In Australia, the average performance engineer’s salary is AU$108,053, or over 5.6 million rupee.

So it’s clear that there are some pretty good reasons for you to become a load tester. What we want to do in this presentation is to walk you through the practical steps to become a load tester. You may not be an expert by the end of the next 30 minutes, but we hope we can convince you that you can get started in load testing whether your background looks like Tim’s or mine.

The recipe for load testing

Tim:

Load testing is like cooking a meal. This is one of my favorite dishes, ALOO MATAR.

You can approach cooking as a science. There are recipes with a list of ingredients, a precise measurement of how much you need for each, and a procedure for creating the dish.

But you can also look at cooking as an art. Cooking the same dish over and over again will give you an experience in working with those ingredients that a novice cook won’t have. There’s room in cooking for adding one ingredient, changing the quantity of another, or just trying something new.

A master chef knows how to use both the science and the art to create something delicious. But you don’t HAVE to be at that level in order to cook.

Anyone can cook. If you know how to follow a basic recipe, you may not create a masterpiece, but if you’re like me and you haven’t had lunch yet, you’ll still end up with something you’ll eat.

So here’s the recipe for load testing.

Nicole:

PEA:

Preparation

Execution

Analysis

P is Preparation. To prepare for a load test, you need to get your ingredients ready. You need to decide what it is you want to test, why you’re testing it, and how. You write load testing scripts that will simulate real users as closely as possible.

E is Execution. This is when you do the actual cooking. You run your load tests and scale them up according to your requirements.

A is Analysis. You figure out whether the dish you created is edible. You determine how your application servers responded during the load test, and decide if that’s acceptable.

Preparation

Nicole:

Load testing doesn’t start with writing a script. Load testing starts with building a foundation for later tests.

Part of that foundation is asking WHY. Why do you want to do a load test? Has there been a performance-related issue before? Are you trying to determine whether an application is ready to go live, or is it already live, and you just want to improve its performance? What type of application do you want to test?

WHOM are you cooking for? I think you’d cook differently for yourself than you would for Sachin Tendulkar. What do your end users typically do on your site? Where do they come from?

These are questions that need to be answered before you think about scripting.

Execution

Nicole:

So now we’re ready to start cooking. This is the Execution phase. You’ll need a machine to use as a load generator. Depending on the number of users you want to run with, you may need more than one machine. You might be tempted to run a load test on your laptop, or on a bunch of machines you have lying around, but that introduces new variables that may affect the results.

Like any good cook, you’ll also need to make sure you have all your ingredients handy. In load testing, that means making sure your scripts and test data files are ready to be executed on every load generator.

Both of these problems – provisioning a load generator and transferring files over to them – are solved pretty easily by running on the cloud using something like Flood.

‍

Analysis

The last part is Analysis.

This is a pretty crucial moment, because this is when you figure out whether what you cooked was delicious or not. Unfortunately, in load testing, it’s not always easy to tell how your test went. It’s like this photo of pakora (onion is my favorite kind). It’s kind of hard to say what expression that is in their faces. Is it disgust? Or is the pakora so yummy that they just can’t wait to eat more?

Load testing typically generates a LOT of data. You’d have to go copy every single file from every load generator you used. But can you imagine the hundreds of load generators that Hotstar used? Of course they didn’t manually copy those files. Instead, they used Flood.

On Flood, we show real-time graphs as you’re running the test so that you can tell immediately whether transactions are failing or whether a server has stopped responding.

After the test, you can also compare this data to previous tests to spot trends, or even create your own dashboards using an analytics engine.

Serving the meal: Continuous Integration

Nicole:

You’ve prepared for a load test, you’ve executed a load test, and you’ve analysed the results from the load test. Congratulations, you’re a load tester! By following this recipe, you’re already doing better than many people, and you’re certainly doing a lot better than those who do not run any load tests at all.

The cool thing about load testing, though, is that this is just the beginning. There’s always room for improvement.

For these DABBAWALLAS, having the meals prepared is not the end of the job. Once the food is ready, it needs to be packed into lunchboxes, picked up, separated according to destination, put on trains, and delivered. Later, the empty lunchboxes are picked up, and the system works in reverse.

It turns out that this is actually a pretty sophisticated system. In fact, it’s been estimated that only one tiffin every two months is not delivered. That’s equivalent to a Six-Sigma quality rating of 99.9999999% (that’s nine nines) or more.

These dabbawallas didn’t get that rating just by delivering quickly. They did it by being consistent and reliable. They did it by having a process.

Once you see the benefits of running a load test, you may also start to understand that the real value of load testing is not in a test that you run once whenever you think of it. Imagine having the benefits of a load test tied to your development process, so that it is automated.

Anyone can be a load tester

Nicole:

So that’s the recipe for becoming a load tester.

Preparation – Execution – Analysis.

In Preparation, you determine why you’re testing and what tool best fits your needs. We showed you how to use Element, but there are many open source tools out there. Don’t get overwhelmed by scripting– a lot of times it can just mean replacing a URL with the one YOU want to test.

In Execution, you run a baseline test that will serve as a point of comparison for future testing, choose the number of users and the number of regions you want, and then you run your test. Running on the cloud is the easiest (and usually cheapest) way to get started.

In Analysis, you view the results in real-time, compare historical performance to see trends, and share reports to stakeholders so that your team can make changes to your application. We also talked about the bonus topic of CI/CD and how you can incorporate load testing into your pipeline to get a continuous feedback during your development cycle.

Tim and I started in very different places and had different experiences before we discovered load testing. But eventually, we ended up on the same path, and that’s what we’ve outlined for you here. We hope that this convinces you that load testing doesn’t have to be complicated.

Tim:

You absolutely don’t need to be an expert to run a load test. Even if you don’t think you’ll get all these steps right, a load test that is less than ideal is still better than no load testing at all.

In the end, anyone can be a load tester with the right attitude and the right tools.

Thank you for listening.