In previous posts we have shown you how to load test using Ruby with JMeter for a typical rails application, and how to tune similar applications using New Relic and Flood.
These are great posts when delving into performance and load testing as a separate activity, but perhaps something you don't do everyday as part of your development, test and release cycle.
In this post we're going to show you how to incorporate load testing using RSpec and some handy integrations with Flood IO and GitHub. We'll use New Relic's Code Kata as our application in much need of performance tuning, and we'll show you how you can use the flood-capybara gem to easily turn your test specifications into load tests on Flood IO. We'll also show how you can track performance improvements you've made with links between Flood IO and GitHub.
Our Setup
Our demo application has been deployed using Heroku. We've chosen RSpec to support a Test Driven Development (TDD) cycle where we can automate browser based tests with a view to running those same tests on Flood IO.
Capybara helps us test web applications by simulating how a real user would interact with the app. We can stitch Flood IO and Capybara together with the flood-capybara gem.
Getting Started
First we'll include the necessary gems in our Gemfile and bundle install.
Then we'll add some basic feature specifications that navigate their way through various pages on the site. It looks something like this:
bash
feature 'Navigation', flood: true do
scenario 'they visit the loop page' do
visit '/loop'
expect(page).to have_content 'The Loop'
end
...
If you're already familiar with feature specs using RSpec and Capybara read on, if not, we recommend reading up on some of the basics.
Running Specs Locally
To run the specs locally we just bundle exec rspec and we'll start seeing results as follows:
bash
$ bundle exec rspec spec
.......
Finished in 20.95 seconds (files took 2.67 seconds to load)
7 examples, 0 failures
Running Specs on Flood IO
To run the same specs on Flood IO we use our API token as follows:
bash
$ flood-capybara spec \
--api_token=$FLOOD_API_TOKEN \
--grid=1QNtoBftrokSErYJdTHRQg \
--rampup=60 \
--duration=120 \
--url=https://flood-newrelic-ruby-kata.herokuapp.com
We can also run this as a rake task e.g. lib/tasks/flood.rake
bash
namespace :flood do
task run: :environment do
system %{
flood-capybara spec
--tag flood
--api_token=#{ENV['FLOOD_API_TOKEN']}
--rampup=#{ENV['RAMPUP'] || 60}
--duration=#{ENV['DURATION'] || 300}
--url=https://flood-newrelic-ruby-kata.herokuapp.com'
}.squish
end
end
The flood-capybara gem will parse specs locally then run the same specs on Flood IO. We will see the following output:
bash
$ bundle exec rake flood:run
Flood specs:
---
- they visit the loop page
- they visit the big list page
- they visit the tweets page
- they visit the cache page
- they visit the async page
- they visit the many assets page
- they visit the errors page
Flood results: https://flood.io/l88JGsxdBZHPHRgFEfFB2g
Other options available when using this gem:
- grid the grid id you want to run this test on.
- rampup the rampup time for the test.
- duration how long you want the specs to iterate for.
- url the URL which Capybara.app_host will get set to.
- tag the tag to filter the examples by.
Initial Baseline
With just 10 users we can already see poor response time performance when running on Heroku using a 2x standard dyno (1GB of RAM).
+12 second response times are shown on Flood IO when running with less than 10 users.
We can also see tell tale signs of poor performance in Heroku logs including Error R14 (Memory quota exceeded) and Error H12 (Request timeouts)
bash
2015-04-09T02:28:21.127912+00:00 heroku[web.1]: Process running mem=1331M(130.1%)
2015-04-09T02:28:21.127912+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-04-09T02:28:25.561647+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="//big_list" host=flood-newrelic-ruby-kata.herokuapp.com request_id=2869ebe5-853c-40b1-b894-bae080aef5b1 fwd="52.64.57.104" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0
As bad as it seems, we call this our first baseline from which we intend to improve. We can set this flood to repeat whenever we push changes to GitHub using webhooks.
Webhooks via GitHub
In the simplest form, we can let GitHub repeat an existing Flood test every time it receives a push event.
Now when we push to master, the flood test is repeated. We apply all the tuning fixes we previously demonstrated and compare the results in Flood IO.
We can now see a 10x improvement in response time performance. We can also jump to the commit history within GitHub from Flood IO in future to see what changes affected this baseline at this point in time.
Where to from here?
If you weren't using GitHub there's nothing stopping you repeating tests in a similar fashion from any of your CI / build automation tools by using the following endpoint:
`POST api.flood.io/floods/:flood_id/webhook`
You might want to differentiate branches such that performance and load testing is only run on develop / staging environments, wait for CI tests to pass green before running performance tests or other such variants. There's plenty of options to drive Flood IO from our API.
You can also run much larger tests. Capybara tests on Flood IO are limited to 10 users (browsers) per grid node but there's nothing stopping you from running floods on much larger grids across multiple geographic regions.