When it comes to self assessment, Captain Riker sums it up:
“The game isn’t big enough unless it scares you a little.”
— William Riker
At Flood IO we take performance seriously and are always looking to improve. We find an increasing number of customers come to us for advice and not just execution of their test plans. A lot of that advice centres around how to improve first visit (empty cache) performance. Luckily you don’t need an expensive tool to figure this out. Tools such as Google Page Speed and Yahoo YSlow offer simple, straight forward advice on how to optimise web page performance.
There’s been a lot of media attention on this lately, just follow the #webperf hash tag on Twitter for example. If you’re not already familiar with the tools, go check them out. They’re easy to install and you can test your own site with a simple browser extension.
Which brings me back to the topic of this post. With all the SaaS performance test vendors out there and barrage of advice being offered, are they dogfooding their own product?
We took a look at 12 vendors that advertise in this space including ourselves to see how we stand up.
We chose from the following SaaS performance test sites to run the comparison tests:
Flood, Loadstorm, Apicasystem, Loader, Telerik, Neotys, Neustar, Loadimpact, Blazemeter, Blitz, Soasta and Smartbear.
We simulated a first visit to the home page of each product with an empty browser cache, using the free service of GTMetrix. This site runs reports from Vancouver Canada using Firefox (Desktop) 25.0.1, Page Speed 1.12.16 and YSlow 3.1.7. Tests were automated using the GTMetrix API and a simple bash script we cooked up for this purpose. This makes the test repeatable and can be expanded to include other sites if interested.
For each dimension of the test we rank sites from best to worst in terms of their individual score. You can view the summary results here which includes links to the individual GTMetrix formatted reports.
Page Load Times (ms)
This is a measure of page load time in milliseconds and is perhaps the most contentious dimension measured. Typical caveats to this score will be things like repeatability (hitting cold or warm servers), geographic distance from test to origin, TCP connection overheads, SSL negotiation and so on. In any case, Loadstorm was ranked number one here, with the majority under 3 seconds.
1639 http://loadstorm.com 2104 http://apicasystem.com 2215 http://loader.io 2585 http://www.telerik.com 2688 http://loadimpact.com 2816 https://flood.io 2876 http://www.neotys.com 3019 http://neustar.com 3153 http://soasta.com 3472 http://blitz.io 3683 http://smartbear.com 3848 http://blazemeter.com
AKA the how bad does my site suck on a high latency mobile network score or more commonly the I don’t know how to combine files and minimise requests score.
80% of the end-user response time is spent on the front-end. Most of this time is tied up in downloading all the components in the page: images, stylesheets, scripts, Flash, etc. Reducing the number of components in turn reduces the number of HTTP requestsrequired to render the page. This is the key to faster pages.
It’s an area in which Flood IO is quite aggressive in terms of optimizations. In fact a second visit to our home page with a primed cache is just 2 requests. We take care to combine assets into single files, use CSS image sprites where possible and generally reduce the amount of HTTP requests required. Very important for high latency links like mobile users and often overlooked by clients including our competitors.
12 https://flood.io 30 http://loadstorm.com 38 http://www.telerik.com 40 http://loadimpact.com 41 http://apicasystem.com 45 http://www.neotys.com 46 http://loader.io 55 http://blitz.io 58 http://neustar.com 75 http://smartbear.com 87 http://soasta.com 110 http://blazemeter.com
Page Weight (B)
AKA the but I thought everyone was on 100 Gigabit Ethernet score.
Put simply, what is the total page weight in Bytes for all content on your page for a first visit with an empty cache.
Apparently web page size has grown +151% in the last 3 years with the average page size now 1575 KB. Flood IO is in the middle of that range with 719 KB and some of the worst offenders creeping above the average. It makes you wonder, just how good do those images need to be as part of your market-eering?
261625 http://loadstorm.com 544241 http://loadimpact.com 545797 http://apicasystem.com 594202 http://www.neotys.com 597154 http://www.telerik.com 654106 http://loader.io 690677 http://neustar.com 719168 https://flood.io 922117 http://blitz.io 1094106 http://blazemeter.com 1574796 http://smartbear.com 1857024 http://soasta.com
Google Page Speed
AKA the how does Google page speed rank my site score.
Google’s Page Speed best practices cover many of the steps involved in page load time, including resolving DNS names, setting up TCP connections, transmitting HTTP requests, downloading resources, fetching resources from cache, parsing and executing scripts, and rendering objects on the page.
It’s a useful comparison between sites because it’s effectively a static analysis against a common set of rules. The higher the score the better. Flood IO came equal first with Loadimpact and the majority of sites score well in the +90% range. Some sites can definitely improve though.
96 https://flood.io 96 http://loadimpact.com 93 http://www.telerik.com 93 http://loadstorm.com 92 http://soasta.com 92 http://smartbear.com 92 http://blazemeter.com 89 http://loader.io 87 http://apicasystem.com 84 http://www.neotys.com 68 http://neustar.com 52 http://blitz.io
Note: this is slightly different to the Google PageSpeed Insights score not listed here, but this also gives you the ability to view your score from both mobile and desktop devices. It’s worth trying for yourself. Flood IO scores an impressive 98/100 for both platforms.
AKA the I don’t believe Google, what does Yahoo think score.
We’ve long been a fan of this browser extension. Some dimensions of the score you might need to take with a grain of salt, and it is generally a little more harsh but once again, as a static analysis for comparison, it’s a great measure of performance. Looks like Flood IO comes out on top again, but no surprise since we focus a lot of our attention in improving these scores.
96 https://flood.io 93 http://apicasystem.com 90 http://loader.io 89 http://loadstorm.com 81 http://www.neotys.com 81 http://loadimpact.com 75 http://blazemeter.com 72 http://www.telerik.com 70 http://soasta.com 68 http://neustar.com 67 http://smartbear.com 63 http://blitz.io
Steps to Improve
So what have we been doing to improve our own score lately? We like to use another free service provided by WebPageTest which lets us test these different performance dimensions in our own time and from different locations around the world. Example results are shown here
- ensured our resources are compressed and served via a CDN where possible.
- minified our resources to the extent we still maintain our core functionality, look and feel that we’re happy with.
- taken steps to improve server response time by removing or tuning back end system calls through application profiling.
- leveraged browser caching as much as possible including correct use of expires and cache-control headers.
- optimised images as much as possible including use of CSS image sprites to further reduce number of requests to the origin servers.
The waterfall chart for our first visit performance is quite telling and gives us ideas for further optimizations, although how much is enough?!
First impressions count, and we hope this post demonstrates that Flood IO is indeed eating its own dog food.