The New Relic design team recently tried out remote user testing for one of our projects. We want to share what turned out to be a valuable process for gathering insights about our product and our users. In this first post, I’ll give you an overview of the questions we hoped to answer and introduce the method we used.
We give our customers access to data which helps them focus their work and make data-driven decisions. In this respect we drink our own champagne, going so far as to use New Relic recursively on New Relic. But sometimes we can’t get the type of data we need from the stats on our own systems. When it comes to the User Experience of our product, we often get to use other tools and learn a lot about our system from a whole different perspective — through the eyes of people using it.
After taking a look at our conversion data, we knew there was a problem. Many people come to our site every day and sign up for an account, but only a portion of them manage to actually deploy New Relic. Installation instructions are the first thing you see after you sign up, so why the attrition?
There were many hypotheses, and we even have a survey we send to people who don’t deploy. These are some of the multiple-choice answers that people confirmed in these surveys:
- My app isn’t ready yet
- I’m too busy
- Someone else at my company will deploy
- Installing the agent was difficult or didn’t work
- I already deployed (perhaps using a different account)
But there are a few problems with just asking multiple-choice questions. The person might have been intimidated by the deploy instructions, or thought it would take too long, or maybe had merely signed up for the free T-shirt, or had an issue with something else we hadn’t thought to ask in the survey.
Adding to the issue, the surveys were sometimes sent a month or more after the customer had signed up; accurately remembering what it was that really stopped them from deploying may have been lost in time. Or they may have just picked the closest answer that was easy, when it was really something more complex.
Ultimately, we didn’t know why people had signed up in the first place. For instance, if someone signed up never intending to deploy, what were they expecting from signing up? In the end we were just left with a lot of unanswered questions.
The Solution: Remote Research
We knew we would get a better idea of what stumbling blocks our users are running into if we watched them as they deploy. Hooray! User research!
One big complication: Even typical recruitment for studies like this can be difficult, but at the center of our research was a question of “who are the people signing up on our site?” How could we recruit people who were actually interested in our product and behaved like the wide range of people signing up for our site if we didn’t fully know who that was in the first place?
We decided to go straight to the source and recruit our own users, live!* Why?
- The right people: We weren’t just interested in the functionality and mechanics of our site. we also wanted to understand the whole thought process leading up to registration, all the way through deploy. In order to learn all of this, we needed to talk to people who were in the same headspace as our real customers: interested in New Relic and motivated enough to register. By recruiting our actual users we knew we had the right audience.
- The right timing: We also needed to capture our users at the right moment, just after they had signed up but before they had gotten too far into the process of installing or had given up. By talking to them right away, we were had a better chance of understanding the real reasons behind their actions.
- The right environment: And finally, because we wanted to see the process when people were able to successfully install (or perhaps tried and failed), we needed to see them in their own environment, using the dev tools already set up on their machines to attempt to deploy New Relic locally, to their dev environment or sometimes even their production environment.
We could never have accomplished all of these with a typical lab study or even a scheduled remote study.
The Game Plan
So how does remote research work? We’ll go into the details in a future post*, but the core idea is pretty simple: we used an online tool, Ethnio to manage the live intercepts and pop up invitations to a subset of our customers after they registered. The invitation asked for an hour of time in exchange for an Amazon gift card, and was followed by a screener questionnaire to help weed out “fakers” and hone in on certain demographics.
When we were ready to start testing, we turned on the Ethnio intercept and called up suitable recruits right after they filled out the screener. Once we got their consent to participate, we typically had them join us in GoToMeeting; that way the user could screenshare with us, and we could record the session.
This allowed us to watch over their shoulder as they took their next steps to get started using New Relic. Since our design team is co-located in San Francisco and Portland, using a remote conferencing solution also allowed my colleague and I to work together, with one of us moderating the interview, the other taking notes.
In Part 2, we’ll talk in more detail about the methods we used to run the user interviews, specifically how we were able to utilize Ethnio and GoToMeeting. Stay tuned!
*Fortunately, we aren’t the first design team that’s run into this issue and we found some great guidance and tools for doing live recruiting:
You can also find more details and options for doing this type of research yourself, online.