How Do You Measure Up? Introducing the App Speed Index

By Posted in New Relic News, Tech Topics, Top Post 25 June 2012

At New Relic, we take pride in what we do. We help our customers monitor their sites and steadily improve the performance of their web apps. But for our customers (as well as ourselves) performance tuning is a lonely endeavor. You tune, you see a speed boost, you celebrate, and you (usually) stop there.

How Fast Is Fast Enough?
Setting performance goals is challenging. It’s easy to find out how fast your site is, but have you ever wondered just how fast your site should be? Ideally, you’d know how fast your competition is and tune accordingly. Or, have you ever had to argue that your site’s performance is a problem? We’ve all read about Amazon’s results that correlate increases in site speed with more sales conversions. Google has found that search traffic is correlated with page load times: more speed equals more traffic. But proving the value of high performance is less clear if you aren’t running an eCommerce site. Wouldn’t it be great if you could see how your tuning efforts compare to your competition?

Leveraging Our Architecture For Your Benefit
The New Relic architecture is built around a SaaS model. That means that unlike many of our competitors, we have access to all of our customers’ data. And we’re using that to your advantage. Our newest feature, the Application Speed Index, looks at your site’s performance based on four key metrics (end-user response time, server-side response time, error rate, and application downtime) and measures them against other sites like yours. Now you have an easy way to see how your performance stacks up against other sites in your industry.

Opting-In: What You Get
To use the Application Speed Index, you opt your app into a particular group, also known as a Peer Group. Each Peer Group represents a different type of application, such as an eCommerce site or a mobile app. Many of our customers use New Relic to monitor multiple sites and the users of each may have very different expectations. So we decided to group along application lines, rather than industry ones. This keeps the comparisons more meaningful and relevant.

Screenshot of ASI signup

You’re probably thinking, “Wait, what? I get to see other peoples’ performance data? What are you doing with mine?!?” Don’t worry, though. We give you complete control over whether or not your data is shared, and your data will always remain anonymous. When you opt-in you’re agreeing to make the end-user response time, server-side response time, error rate and downtime of your app available for comparison. Others in your Peer Group have done the same, but you’ll never know exactly who they are.

Under the Hood: How It All Works
Each night around Midnight PT, we calculate summary records for the previous day’s performance for all apps that have opted in to the Speed Index feature. We take the average end-user response time, response time and error rate, as well as the total downtime for the day. Once we have all of the summaries, we rank them and calculate percentiles.

Let’s Get To the Data
When you visit the App Speed Index report for your app, you’ll get a view of your Peer Group’s performance over the past thirty days. We’ll tell you your rank, both absolute and in percentiles, within the group for the four key metrics we’re tracking. You’ll also see the distribution for those four metrics. Not only will you know what your rank is, you’ll see how much you need to improve in order to move up! We’ll also show you your day-to-day average performance and ranking for the past thirty days, so that you can see how you’ve gone up or down, and how that has affected your rank.

Screenshot of an ASI report

The ranking has one minor detail you should know about: we honor ties. For four apps with response times of 1s, 1.2s, 1.2s and 1.5s, the ranks will be 1, 2, 2 and 4. Since there were two #2’s, we drop the #3. This is particularly important for the availability data since most sites have no downtime. We’ve also learned that most sites have no errors (go devs!) and so it’s important there as well. For these two metrics, any move away from rank #1 is a very big drop. So watch that first step — it’s a doozie!

Screenshot of speed index infographicLeaderboards: Find Out Who’s Winning and Get Your Brag On!
Remember when I said that you would remain anonymous? That was mostly true. It’s certainly true if you want it to be. But if you have a lightning fast site and want extra bragging rights, you can let us publish your name in our top-ten leaderboards. Each day, we publish a list of the top ten performers in each Peer Group. For this ranking, we only look at the end-user response time, which is the single most important metric used in this feature, since it captures your users’ experience on your site. You can control these settings at the time you opt-in and on the app speed index page — just look for the box on the right-hand side.

This is the first time we’re publishing the results from our big date in the form of a live-data infographic, that’s currently in beta and can be found here.

As an added bonus for being in the top ten, we’ll link to your site so everyone can experience your blazing speed.

Minimum Requirements: In the Interests of Fairness
When we were planning this feature, we thought a lot about what would make for a fair comparison. For example, an app that is only responding to API requests should have an average response time of under 50ms. Something shorter like 10 – 20ms is normal. For an app that’s rendering full HTML page views, a 20ms response time would be nothing short of miraculous. So it doesn’t make sense to let API apps in to the same groups as user-facing apps. In fact, we decided that end-user data is required for getting in to a peer group to see your Speed Index.

We also thought about staging apps and the like. We have our own and we didn’t want them to be included in the groups. Because staging environments don’t see the kind of traffic that production environments do, and they frequently aren’t running on super-charged hardware, we thought that we’d exclude those from the groups as well. To enforce that requirement, we have a 10 ppm minimum end-user throughput in order to opt-in.

Stability Has Its Benefits
Some of our customers might have a hard time putting their primary apps into a single category. After all, many startups are innovators, creating products that don’t fit into standard categories. If you have an app like this, you might be tempted to shop around and move between groups a lot. We hope that you’ll take a look around and then settle into one group. One of the categories should speak to you and your application more than the others. To encourage you to stay there, we’ll hold on to your daily Peer Group data forever as long as you stay in that group. So if you’re in a group for a year, you’ll have a year’s worth of historical data sitting around. The day you leave the group, it will all disappear.

Getting Your Info
You can visit your app’s Speed Index report at anytime to see where you stand. We’re also adding the info to our weekly emails, so you can stay in touch with how your Speed index is trending. If it moves up or down a lot, you can then use that to make prioritization decisions for your work.

Helping You Help Yourself
We want you to succeed. Our mission at New Relic is to help you make your site better. We hope that armed with hard data about other apps like yours, you’ll be able to make smart decisions about what to tune, how to make your users happy, and when you’ve tuned enough and can work on features instead. Either way, your users win. When they win, you win. And that makes us happy.

Try New Relic for free today.  Sign up and deploy and instantly gain deep insight into your web apps.

brent@newrelic.com'Lead software engineer

Tell us your thoughts Or Send us an internal high five

Talk to @newrelic