JMeter is not a Browser
A common misconception of first time users of JMeter is that it functions exactly like a browser. It doesn’t. JMeter like most performance test tools can emulate browser behaviour in a number of ways. Static resource fetching, caching, cookie and header management are some of the main components to browser emulation. Let’s talk about caching and how that can be implemented in JMeter.
You’re probably already familiar with your browser’s local cache. This lets you keep representations of content on local storage. When you hit your browser’s back button for example, chances are you’re seeing cached content. Based on simple rules, the browser cache typically checks representations are fresh according to the current browser session. If representations are determined to be stale, then new requests will be made to the origin server.
Other caches, such as proxies, can exist between your browser cache and the origin server. It’s important to be aware of these as intermediary caching obviously impacts the workload profile of the origin server itself. Often these are outside of your control, and are particularly prevalent in any form of testing outside of a controlled network environment.
Freshness and Validation are two important concepts of cache management. Fresh content is typically served straight from the browser cache, whilst validated content will avoid requesting the same content if it hasn’t changed. Cache management, in terms of determining if content is stale or fresh, is mostly implemented via HTTP headers.
Pragma headers specify optional behaviour. The Pragma: no-cache header is only defined for backwards compatibility with HTTP/1.0 and is equivalent to the Cache-Control: no-cache header directive. Some caches will simply ignore the Pragma directive so it’s not a great way of ensuring content is not cached.
Cache-Control headers were introduced in HTTP/1.1 to specify directives that must be followed by all caching implementations in the request/response chain. The common Cache-Control directives you will come across are:
The Expires header gives the date/time after which the response is considered stale. The presence of an Expires header field with a date value of some time in the future on a response that otherwise would by default be non-cacheable indicates that the response is cacheable, unless indicated otherwise by a Cache-Control header field.
The Last-Modified header field indicates the date and time at which the origin server believes the variant was last modified.
The ETag header field provides the current value of the entity tag for the requested variant.
JMeter Cache Manager
The JMeter HTTP Cache Manager is used to add caching functionality to HTTP requests within its scope. Essentially it will check response headers and respect the majority of headers related to cache management around Expires, ETag and Cache-Control directives.
It implements a per-user/thread Map which has a maximum size and uses a Least Recently Used algorithm to remove items from the Map when the maximum size is reached and new items are added.
The default size is 5000 items, an item being stored as the URL along with Last Modified, Expires and ETag header information. It’s possible to increase this value, however setting an appropriate value for this is difficult with no feedback from JMeter itself. Flood IO provides a Response Code timeline chart, so if you only see response code 200, or a much higher rate of response code 200 compared to expected response codes 304 and 204 when using the Cache Manager you may need to adjust.
Expected Response Codes
When using the JMeter Cache Manager you can expect to see HTTP response codes 304 or 204.
When JMeter makes a conditional GET request but the document has not been modified, a server will typically respond with a 304 response code and no response body.
Thread Name: Thread Group 1-1 Sample Start: 2013-10-29 20:20:07 EST Load time: 3 Latency: 0 Size in bytes: 150 Headers size in bytes: 150 Body size in bytes: 0 Sample Count: 1 Error Count: 0 Response code: 304 Response message: Not Modified
When JMeter simulates using content directly from its cache, it will record a 204 response code and no response body.
Thread Name: Thread Group 1-1 Sample Start: 2013-10-29 20:20:07 EST Load time: 0 Latency: 0 Size in bytes: 0 Headers size in bytes: 0 Body size in bytes: 0 Sample Count: 1 Error Count: 0 Response code: 204 Response message: No Content
Confusingly, this does not mean the server has fulfilled the request nor responded with an entity-body, in order to return updated meta information as per the HTTP/1.1 specification. The HTTPClient4 Implementation in JMeter uses a constant from the HttpURLConnection which sets this value. So rest assured, no actual requests are made to the origin server in this case.
Testing the Directives
Putting this all together, we can test a simple scenario which has two iterations with a Cache Manager, to simulate the first visit (with an empty browser cache) and a second visit (with a primed cache). The target site is a simple stub using Ruby / Sinatra to exercise different cache control headers.
If a no-cache directive is set, JMeter will keep this in cache, but set the expires date to null which will trigger a revalidation for each request. You will generally see a HTTP response code 304 for this type of conditional request. This is consistent with the HTTP/1.1 specification which must NOT use the response to satisfy a subsequent request without successful revalidation with the origin server.
If sent in a response, a cache MUST NOT store any part of either this response or the request that elicited it. JMeter does in fact cache this type of response so it is probably not consistent with the HTTP/1.1 specification.
This has since been fixed via bug ID 55721 thanks to prompt support by the JMeter core team!
The remaining directives including Expires / ETag headers all behave consistently with the HTTP/1.1 specification. In general, if the “Use Cache-Control/Expires header” option is selected in the Cache Manager, JMeter will issue a HTTP response code 204 for content in its cache, otherwise it will make a conditional request and you will see a HTTP response code 304 where appropriate.
A Real Example: squarefoot.hk
Let’s look at a realistic example using a real estate site based in Hong Kong. This example is interesting because it has a wide range of content and is fairly heavy in terms of a first visit to the site.
YSlow shows that the page has a total of 136 HTTP requests and a total weight of 933.6K bytes with an empty cache.
Subsequent visits to the site should see 17 requests with total weight of 165.1K bytes with a primed cache.
Using a simple JMeter plan with a HTTP Cache Manager, 1 thread, 1 iteration with 2 requests to the home page produces the following summary statistics when sampled via a debugging proxy (Charles):
First Visit Empty Cache
Responses 1.03 MB
Second Visit Primed Cache
Responses 123 KB
Not too bad in terms of anticipated drop in requests and response size, however not the same order of magnitude we observed with the simple YSlow comparison.
What this means is that you will need to determine which resources were not downloaded using this approach and include them manually in your test plan if they are significant. There may also be 3rd party domains which we don’t want to test e.g.:
http://b.scorecardresearch.com http://reagroup.122.2o7.net http://secure-sg.imrworldwide.com
That’s why it’s extremely useful running your JMeter test plans through a debugging proxy like Charles or Fiddler to get an idea of what your script is doing for a single user over multiple iterations. You can also do this more simply with a View Results Tree listener in JMeter, but the waterfall view is particularly nice to inspect network traffic.
The other reason for discrepancies between first and second visits to the site is that the contents regularly change. Different images are displayed each time we visit the site so those items won’t already be cached, hence a new request to the origin server. In the case of YSlow, it is comparing the exact same request/response chain.
The JMeter HTTP Cache Manager is used to add caching functionality to HTTP requests within its scope. Essentially it stores a copy of the URL along with Last Modified, Expires and ETag headers taken from the response in a Hash Map which will evict items based on a Least Recently Used algorithm.
It will typically record HTTP response codes 204 or 304 for items which are cacheable and meet freshness or validation criteria. Response body will be empty for these items so you need to be careful with any assertions on content.
It’s extremely useful running your JMeter test plans through a debugging proxy like Charles or Fiddler to get an idea of what your script is doing for a single user over multiple iterations. Alternatively use a View Results Tree listener in JMeter.