We love it when one of our 6,000+ customers writes a blog post to share how we helped them discover a performance problem. Below is a repost (with permission!) from one of our customers – Martin Dell of Kyero.com. You can find the original post here.
New Relic Strikes Again
Just over a week ago, we deployed a totally revamped Kyero.com.
The deployment went well and we’ve been running on the new code since the 13th.
One of the anticipated side benefits of doing this was an increase in performance – faster page load times, and less waiting about for stuff to happen.
Unfortunately, the reverse seemed to be the case. We managed to increase page load times and slow the site down.
One of the problems was that we’ve changed just about every single line of code on the site – so how were we going to find out where the problem was?
I’ve mentioned New Relic before – it’s the performance monitoring tool we use to keep an eye on Kyero.com.
Once again, New Relic enabled us to identify and fix the problem. Here’s a performance graph before and after the fix.
The area to the left of the chart (1) is exhibiting the performance problem we were seeing. We were averaging around 1 second page loads – due mainly to the thick blue icing at the top of the chart (memcached).
Memcached is one of the ways we speed up delivering pages to visitors but here, because memcached was taking too long, it was loading up our servers causing visitors to wait – as indicated by the green ‘queue wait’ section.
The twin peaks of section two are when we deployed a fix for the problem – so performance went slightly screwy for a couple of minutes while that took place.
Section three shows the kind of performance we achieved after deploying the fix. Memcached time is greatly reduced which has, in turn, reduced the queue wait times.
We’ve managed to reduce page load times by around 50% – and we still have some more tweaking to do.
What was the problem with memcached? We use two levels of cache “above” our database to speed thing up.
The fastest is “local in memory”, it’s like having a file on your desk right in front of you.
The next level is memcached, which is still fast but goes over the network between our servers. It’s like having a file in your desk drawer – and you have to open the drawer to find the file.
Our recent deployment stopped the “local in memory” cache from working properly, so instead of getting the file out of the drawer once and leaving it on the desk, we were taking it out, then putting it in, then taking it out over and over again.
Once again, hats off to New Relic for helping us pinpoint a complex problem, and hats off to Chris (one of our developers) for turning that information into an elegant and effective fix. 🙂
Martin Dell, Kyero.com