As members of the New Relic Docs team, we’re passionate about words and language. But, like everyone who uses New Relic, we’re also wild about data.
We gather a lot of data about our documentation site and use it to answer functional questions about our work. We want to know if our docs are helping our customers, if revisions and edits are improving the customer experience, and if we’re prioritizing our work for maximum benefit. To answer these questions, we need incisive, insightful data.
To get that data, we harness the power of New Relic in a variety of ways to help ensure we’re providing the best experience for our customers as we possibly can.
“Was this page helpful?”
The easiest way to find out if our docs are helping our customers is to ask them directly. So, we do exactly that with a form we embed at the top of every page:
Our content management system (Drupal) captures the Yes and No responses and any comments that our customers share. These responses are then sent to New Relic Insights, where a dashboard displays the tally of responses and the overall customer satisfaction rating for our docs, which we track on a weekly basis.
If there is a change in our customer satisfaction rating, we investigate which docs are getting the most responses, how those docs are rated, and then target the low-rated docs for improvements.
For example, our .NET troubleshooting doc recently received some negative satisfaction ratings. To see if we could improve its usability, we did a simple revision to restructure the order of the information in the doc. After publishing the revision, we monitored that doc on our Insights dashboard and saw its rating improve. Because we can track the difference one revision makes, we’re able to make big differences with small changes.
Using customer satisfaction feedback to inform improvements is key to keeping our content helpful. Tracking the satisfaction rating helps us with staffing decisions, and makes it easier to demonstrate to stakeholders throughout the company how good, consistently updated documentation reduces the burden on our support organization.
Keeping the feedback alive
Because feedback from our customers is so important, we also use New Relic to monitor how customers are interacting with our site so we can make sure that feedback keeps coming in. Another Insights dashboard shows us how often customers dismiss the satisfaction form without responding. We compare these dismissals to the number of page views and to the total number of responses, to see how the amount of feedback we get varies over time.
Recently, we ran an experiment to see if we could get more responses if we moved the satisfaction form to a different spot on the page. We did some A/B testing of the placement of the form and tracked the results in Insights:
As you can see, the form got far more responses at the top of the page than at the bottom.
Troubleshooting the docs site functionality
He used New Relic Browser to page through the error details, where he saw the same error (Reference Error: Can’t find variable: Drupal) generated on several API docs pages.
The engineer tested a fix in the docs site development environment, and once he saw that it worked, pushed it live to production. Poof! The errors disappeared.
Prioritizing our work
New Relic Browser shows us a picture of the traffic on our docs site so we can focus on docs that will have the biggest impact for the most customers. We use Insights to track the number of views that our docs get, as well as which docs our customers view the most. We can further segment that traffic information by a set of docs, or even by a single doc, so we can prioritize work on high-traffic docs and features. This helps us understand the resources we might need for future work.
For example, thanks to Insights we know that docs that introduce new products and features consistently get high traffic, so we’re always sure to devote a writer’s time to intro docs. Similarly, we see a lot of traffic on our Introduction to New Relic page, so we work hard to keep it up-to-date as we introduce new features and functionality.
Getting the help we need
The docs at New Relic are collaborative—any New Relic employee can contribute to them. The edits and efforts of our colleagues in New Relic’s support, engineering, and product organizations help our docs stay accurate and up-to-date, even as we ship software updates as often as several times a day.
The Insights dashboard below shows the number of docs pages that were edited in a particular week, and even separates edits made by the Docs team and edits made by subject matter experts and other contributors throughout the company:
Because New Relic helps us track both the work we’re doing and the contributions made from outside the team, we know whether or not we have the resources we need to provide great experiences with our documentation. And when we see increases in contributions from other teams, we know that our documentation training is working.
Enhancing the docs—any way we can
That covers only a few of the many ways that New Relic products help us enhance our docs. We also use New Relic to:
- Make our docs easier to find. We track the search terms our customers use to find information on our site, and we revise our docs, where appropriate, to use the same terms.
- Give our customers the kind of help they need. We can see how often our customers copy code samples in our docs, and we use that knowledge to make informed decisions about the types of content and examples we supply.
- Fix what’s broken. We’ve configured New Relic to create reports of broken links, so we can find them and fix them.
- Meet our customers on their terms. Because we know that some of our customers access our docs from mobile devices, we can monitor the flexible, responsive experience we provide.
- Catch problems early. New Relic Alerts is a great tool to monitor a documentation site. Alerts monitors the status of the site and lets us know when something isn’t working, so we can fix any issues before they affect our customers.
Finally, if you’re as excited about customer feedback as we are, and you want to know more about how we capture and use it to continually improve our site, check out this presentation by Betsy Roseberg, a senior technical writer on our team.