Measuring the Performance Impact of New Features with LaunchDarkly and New Relic Insights

Guest author Patrick Kaeding is lead software engineer at LaunchDarkly, which helps software teams launch, control, and measure their features.

Adding features to your application is great! Users love it, since it makes the app more useful. An extra bullet point on a list of features makes it easier to sell to new customers, and developers love to see people find their products valuable. But new features can also be risky, as changing anything has the potential to cause regressions in existing functionality, and new features can have a negative impact on non-functional requirements, like page load time or other performance metrics.

One way to mitigate the risk is to use “canary launches,” in which you roll out the new feature to a subset of your users, and monitor it for problems. LaunchDarkly is a tool for managing canary launches, and makes it easy to roll out the feature to more (or fewer) users, or to turn it off completely.

New Relic integration

By integrating LaunchDarkly with New Relic, you can measure the performance impact of new features to quickly determine what actions you need to take. To see how LaunchDarkly works with New Relic, let’s look at MapCustomizer, a side project I created that lets users create shareable maps showing multiple locations using Google Maps.

map customizer

 

One of the app’s most commonly requested features is driving directions. But I worried this might be expensive to compute. In the spirit of lean development, I decided to just compute the directions for some maps, but not display them to users.

To find out the effect of adding directions to MapCustomizer, I used LaunchDarkly to tag whether or not each request included the directions calculation and New Relic Insights to capture the performance timings.

How to control a canary launch

Assuming you have LaunchDarkly set up in your application, the first step is to wrap the new feature you are testing with a feature flag (GoogleMapsHelper.getDirections is a wrapper function I wrote around Google’s DirectionsApi):

 

private def calcDirections(map: GMap, user: LDUser): Unit =
  if (map.points.length <= 8) { // exclude overly-large maps from the experiment
    if (ldClient.toggle('enable.directions', user, false)) {
      GoogleMapsHelper.getDirections(map.points) // do nothing with the results for the experiment
    }
  }

 

Once you have the feature gated by the feature rollout flag, you can control the rollout on the LaunchDarkly dashboard, as shown here:

launchdarkly chart

 

I started the rollout at 10%, and when everything seemed fine, I turned it up to 50% to run a more meaningful test. If something had gone wrong, I could have used LaunchDarkly’s on/off switch (aka the “kill switch”) to turn off the feature for all users instantaneously, without having to redeploy.

Viewing results

Since I already had New Relic monitoring my application, I already had request timings for all requests. But now LaunchDarkly tags each request with a value for the enable.directions flag. I didn’t need to do anything special to enable this—LaunchDarkly automatically determines if you have New Relic installed, and annotates the transactions automagically.

With a simple NRQL query to New Relic Insights, I can see how the response times differ between the variations:

SELECT percentile(duration, 90) from Transaction FACET `enable.directions` SINCE 1 week ago TIMESERIES AUTO

 

launchdarkly chart

 

If I’d rather see some plain summary numbers (rather than the nice time-series chart), I’d simply query:

SELECT percentile(duration, 90), count(duration) from Transaction FACET `enable.directions` SINCE 1 week ago

 

launchdarkly chart

Conclusion

Imagine if you could do this with every feature you want to add to your apps! You’d never have to worry that a new feature would ruin the experience for all of your users, forcing a rollback deploy. If New Relic revealed an issue, you could just hit LaunchDarkly’s kill switch and disable the feature until you fixed the underlying problem. Or if you notice that the new feature causes performance issues, you could release it to a subset of your users while you either provision more server hardware or optimize the code to improve performance. In the meantime, at least some users could still benefit from the new feature, a far better approach than holding back the feature from all your users.

LaunchDarkly lead software engineer Patrick Kaeding was a senior developer at Atlassian, where he helped build the Atlassian Marketplace and worked on the Bitbucket team. He has a B.S. in Computer Science from Northeastern University. View posts by .

Interested in writing for New Relic Blog? Send us a pitch!