Do you do load testing and benchmarking of your site before deploying it to production? One hopes that you do, but what URLs are you testing? Do you only test valid URLs? Do you test URLs which you know will fail?
Having access to data being reported from a large variety of different applications, here at New Relic we get to see some interesting situations at times. Sometimes one can distill useful examples of bad application behavior, which may highlight a use case or circumstance you may not have considered.
Once case in point is the performance of a web application in circumstances where a URL does not identify a valid resource. You may well have overlooked this in your testing and given no thought to it, yet it could be unduly contributing in an adverse way to overall site performance.
To illustrate the potential for problems, let us look at an example from our Python agent and the support it provides for the Django framework.
When using the Django framework, Django provides a default handler for dealing with a page not being found when the URL resolver is run or when a user generated Http404 exception is raised by a handler. The name of this default hander shows up as the path ‘django.views.default:page_not_found’ in the web transactions overview.
For this example, when we look at the web transactions by time we find that it is the second largest contributor.
However, when jumping over to look at web transactions by throughput, we find that it is not being called that frequently, only in the order of 2 percent.
Why then is it factoring so high in the list of web transactions based on time? Hover over the entry in the list and we can see that it has a quite large spread of response times.
Looking at the history for that specific web transaction over a period of 6 hours we can see that the high response times aren’t happening all the time though. You may think therefore that it isn’t too bad and you can live with it.
What you have to keep in mind though is that you don’t completely control when users may hit a URL which is going to result in a page not being found. It may be happening infrequently now, but what if an incorrect URL gets included in a page which suddenly becomes popular and users try and follow that link. What if the page already existed and the link worked and you changed your site and broke the link.
What you may see as being an infrequent occurrence could become an onslaught quite easily and if an invalid URL can have quite high cost then the affects could be significant.
Stepping back for a moment and thinking about why we might get ‘Page Not Found’ we come up with two cases.
The first is that the URL is simply not matched by the URL resolver. In this case the problem would be identified very quickly and fall through immediately to the default handler. We still have to render a page indicating that the requested resource could not be found however and this is where the bulk of any time is likely to be consumed for this case.
The second case is that a REST style URL interpretation was occurring and the URL was partially resolved by the URL resolver with the request then being handed off to a handler to resolve further based on parameters appearing as part of the URL. The operation then performed on those parameters may fail to match and as is the convention with REST style URLs a Http404 exception is raised to generate a 404 and flow again falls through to the default handler.
With those two possibilities in mind let us now look at the performance break down for the web transaction.
What we see here is that the majority of the time is being spent within a database query.
Now although it is quite possible that in rendering a response for page not found you might decide to generate some dynamic content, it is probably unlikely. This therefore means it is more likely the query was executed from the actual target handler. Where we have a slow transaction trace we can confirm this.
So the database query was indeed in the target handler with a Http404 exception being raised and control then flowing through to the subsequent default handler for page not found.
Why was the database query so slow then? The problem with database queries is that although they can be fast when the item you are looking for exists, if it doesn’t exist then in the worst case you could trigger a full table scan. If a table contains a lot of data the time involved to do this could be significant.
To resolve this one would need to start looking at the database and what indexes exist on the tables the query is touching. Database explain plans for the specific query can also be useful in understanding what is going on.
In the end you may find you need to optimize database queries and/or add table indexes to ensure not only the successful case performs adequately, but also the failure case where the requested data does not exist. Don’t cover both bases and there is a risk, albeit small, that when hit with requests trying to find non existent data that performance could suffer for no useful outcome.