How to Support 5,000 Customers with No Support Staff

By: Lew Cirne | Posted in: Company News, Jul. 29th, 2010

Last week, New Relic hit another incredible milestone: in just two years on the market, RPM has skyrocketed to 5,000 production-deployed customers around the world. 5,000 customers in 2 years. That’s amazing to me! My previous company, Wily Technology, took 8 years to reach 500 customers, prior to its acquisition by CA. This time around, we are clearly we are doing things a little differently in tackling the app performance management problem. One particular difference is how we support such a large, diverse customer base.

This morning, I looked at New Relic’s support queue and dashboard (courtesy of Tender, a slick SaaS customer support offering brought to you by the great folks at ENTP, a longtime New Relic customer.) I saw some pretty good metrics for such a large customer base: 14 Pending Issues, 1 of them pending today, and an average response time of 10 hours. We are continuing to work on our average response time and would like to see that be lower. But when you think about it, same-day service across the entire customer base is pretty good, given that the majority of our customers are getting free support for RPM Lite, a free product.

New Relic's Support Queue

It will probably surprise you to find out that we actually have no dedicated support personnel inside of New Relic! Each and every support request is handled by a development engineer who has full access to our entire source base, and the full authority to do whatever is necessary to make things right. This includes agent tweaks, production patches and – on rare occasions when we mess up – free upgrades to RPM Gold. Our engineers also have an after hours on-call rotation for urgent issues raised by our Gold customers. So our Gold customers can call any time, day or night, and get a New Relic engineer on the phone within an hour.

So how can a small company of 32 people support 5,000 customers with no support engineers? In a nutshell, it’s about product quality. Not just fewer bugs, but a more intuitive product that’s easier to use and understand. If the product just works, the customer won’t need to reach out to support in the first place.

We have our engineers take support calls because engineers like to code their way out of the support queue rather than execute manual repetitive tasks. Instead of coming up with one-off workarounds for specific customer issues, our team is more likely to implement solutions that the entire customer base can benefit from. Some of our best product ideas come out of support cases. If we can avoid future support issues by adding a feature or fixing a bug that makes the product easier to install, use, or manage, then everybody wins.

Also, our customers like the fact that they are working directly with some serious technical talent. We will never ask our customer if they restarted their computer as our first step towards problem triage.

We also see yet another benefit of being a pure SaaS provider. 95% of our source base is running in production in a single data center over which we have almost complete control. So we don’t need to wrestle with figuring out why a customer is having a problem with a data collection server in their environment. Or if another customer has trouble making sense of data that the RPM user interface is presenting, they can send us a link and we can look at the exact same page. In my previous company, both of these issues would require either an on site customer visit, or months of back-and-forth emails of log files and screen shots to isolate and address the problem. Today, these types of problems are resolved in minutes, and usually benefit the entire customer base as soon as we push to production.

At some point, New Relic will be a large enough company where it makes sense for us to have a separate, dedicated customer support team. It’s hard to imagine our developers handling every support request from a customer base of 50,000 or 100,000 customers. (Yes, we set our sights high!)

Until then, I think it’s great that our development engineers have responsibility not only for writing great software, but supporting it too. In the long run, I think it makes for happier customers and a better product.

Tweet this!

Comments

RSS feed for comments on this post.

14 pending issues is indeed a small number. You did not mention another metric: the number of issues closed per month, qtr, ytd or year. Do the 14 open issues represent 5% of the total received this year?
One way to support many customers with few staff is to receive few support requests in the first place. Your Knowledge Base in Tender contains more than 60 articles. I suspect that many issues are solved via customer self-help before they become support requests.
Because of its searchable Knowledge Base, we also chose Tender for our app (realized-app.com).
Posted: 30 July 2010 at 6:46 am by Gary Fleshman
Great post! Software is a great example of a product that can be best supported by the people who built it because they have the greatest knowledge and ability to make real changes to the product. There are many examples of companies that require all new staff to spend time supporting customers (Freshbooks, Zappos) because this gives them the best understanding of real issues.
There’s a danger of engineers existing in isolation and not solving real world customer problems. If a support query keeps coming up then you can either answer it by robotic stock answers, or solve the actual problem. We did just this and wrote about it at http://blog.boxedice.com/2010/05/28/customer-support-as-product-development-disabling-auto-scaling-server-monitoring/
It’d be interesting to hear how you balance the support queue around your team and how you manage internal response time targets so as not to keep distracting developers from their other tasks. We’d all love response times in the minutes but that’s not always practical without dedicated staff.
Posted: 30 July 2010 at 7:22 am by David Mytton
Great post Lew, while we are a much smaller company, we take the same approach at Olark (http://www.olark.com). Every member of our team does a rotation on support. I think the key issue is to make sure that 1) everyone in the company knows why they have a job [hint: it's the customers], and 2) An engineer with full commit access rarely needs to escalate anything (so quicker service).
I’ll definitely start tracking new relic, looks like you guys are doing amazing things.

lewcirne Reply:
July 30th, 2010 at 12:01 pm
Thanks very much, Ben. I look forward to checking put olark.com.

Posted: 30 July 2010 at 7:41 am by Ben
Thanks for sharing! May I ask how many support requests you get per month? We’re a SaaS company with a pretty similar customer base, we have about 4,500 active customers. We do get about 150 requests per month, support & feature requests / general feedback combined. Just curious how we compare, would be great if you could share some more numbers!
Posted: 30 July 2010 at 7:48 am by Julia
I agree, this is a great post. You summed up our dream as well. We’re actually not an SaaS company, but a traditional printing / manufacturing company that’s trying to find it’s way online.
In our case, most of our team is focused on design-oriented and we all do support. Every inquiry gives us a chance to think about how our web site is failing. Once you hire dedicated support staff that drive to find and diagnose usability problems diminishes. It’s also just plain fun to interact with customers.
It’s very cool that companies are getting comfortable bragging about not having support staff. Years ago that idea would sound insane. Hopefully we’ll be lucky enough to say something similar in a few years. We’re still too young to feel that much pain.
Posted: 2 August 2010 at 10:35 pm by Anthony
Couldn’t have put it better myself – this is pretty much our approach. It is working well enough that I don’t think we’ll ever have a dedicated support team, “You wrote it, you support it” seems to create just the right incentives. We seem to have a situation where a support incident almost always generates an internal Jira ticket to fix the problem at source, which is excellent.
One crucial point though… you have to let the developers have the time to fix the problems they are uncovering, or at least entertain a discussion about those fixes. Sometimes we jointly decide that it isn’t worth engineering ourselves out of a support issue, but everyone buys in to that. Most of the time they’ll put the fix in almost straight away, and we build slack into our development schedules to allow for both the time they spend on support issues and also the inevitable fixes.
I don’t think it will work if you have a very rigid planning system or little slack in the development process, but if you do it works a treat.
Posted: 4 August 2010 at 6:53 am by Simon Coles
Last time I reported a bug, I was told by an engineer that I should write a fix for it and submit a patch.
It’s the response I would expect from an open source project, but not a paid service.

lewcirne Reply:
August 4th, 2010 at 3:19 pm
Hi John,
I’m very sorry that you had this experience. If the issue you had was indeed a bug then we should take responsibility for fixing it. When we receive a feature request for our agent, or a request to support a new environment we don’t currently support, then we often encourage people to submit improvements to our community-contributed agent and we’ll make efforts to incorporate that goodness into our supported agent in due course.
If you feel that the issue is a legitimate bug, please forward any information you have about it to my attention and we’ll have a look at it. My email address is lew at (our domain name)

Posted: 4 August 2010 at 2:59 pm by John
Great article. You forgot to mention the free lunch plan for customers who all already got all NewRelic they can have.
Posted: 20 August 2010 at 10:36 am by rubyorchard
I’m curious, does your support staff have “write” access to production data? I know, unusual question, but I currently work in a culture where the support staff demands “write” access as the only way they know to support our programs and they have been doing it for years. I’ve been using the same argument that “developers like to code their way out of the support queue rather than execute manual repetitive tasks”. So let the developers support these requests and we’ll get rid of silly manual repetitive tasks and hopefully reduce the number of support requests/issues! thanks.

lewcirne Reply:
February 22nd, 2011 at 1:03 pm
Hi Cory,
In general, our philosophy is to empower the individual contributor as liberally as possible so that they can solve a customer’s issue as rapidly as possible using their own judgement. So yes, our support and development engineers have write access to the production environment to do things like update customer account status or turn on free gold access for a customer who has had a bad experience. We remind our employees of that old saying “with much power comes much responsibility” and trust them to do the right thing.

Posted: 22 February 2011 at 5:52 am by Cory

14 pending issues is indeed a small number. You did not mention another metric: the number of issues closed per month, qtr, ytd or year. Do the 14 open issues represent 5% of the total received this year?

One way to support many customers with few staff is to receive few support requests in the first place. Your Knowledge Base in Tender contains more than 60 articles. I suspect that many issues are solved via customer self-help before they become support requests.

Because of its searchable Knowledge Base, we also chose Tender for our app (realized-app.com).

Posted: 30 July 2010 at 6:46 am by Gary Fleshman

Great post! Software is a great example of a product that can be best supported by the people who built it because they have the greatest knowledge and ability to make real changes to the product. There are many examples of companies that require all new staff to spend time supporting customers (Freshbooks, Zappos) because this gives them the best understanding of real issues.

There’s a danger of engineers existing in isolation and not solving real world customer problems. If a support query keeps coming up then you can either answer it by robotic stock answers, or solve the actual problem. We did just this and wrote about it at http://blog.boxedice.com/2010/05/28/customer-support-as-product-development-disabling-auto-scaling-server-monitoring/

It’d be interesting to hear how you balance the support queue around your team and how you manage internal response time targets so as not to keep distracting developers from their other tasks. We’d all love response times in the minutes but that’s not always practical without dedicated staff.

Posted: 30 July 2010 at 7:22 am by David Mytton

Great post Lew, while we are a much smaller company, we take the same approach at Olark (http://www.olark.com). Every member of our team does a rotation on support. I think the key issue is to make sure that 1) everyone in the company knows why they have a job [hint: it's the customers], and 2) An engineer with full commit access rarely needs to escalate anything (so quicker service).

I’ll definitely start tracking new relic, looks like you guys are doing amazing things.

lewcirne Reply:
July 30th, 2010 at 12:01 pm

Thanks very much, Ben. I look forward to checking put olark.com.

Posted: 30 July 2010 at 7:41 am by Ben

Thanks for sharing! May I ask how many support requests you get per month? We’re a SaaS company with a pretty similar customer base, we have about 4,500 active customers. We do get about 150 requests per month, support & feature requests / general feedback combined. Just curious how we compare, would be great if you could share some more numbers!

Posted: 30 July 2010 at 7:48 am by Julia

I agree, this is a great post. You summed up our dream as well. We’re actually not an SaaS company, but a traditional printing / manufacturing company that’s trying to find it’s way online.

In our case, most of our team is focused on design-oriented and we all do support. Every inquiry gives us a chance to think about how our web site is failing. Once you hire dedicated support staff that drive to find and diagnose usability problems diminishes. It’s also just plain fun to interact with customers.

It’s very cool that companies are getting comfortable bragging about not having support staff. Years ago that idea would sound insane. Hopefully we’ll be lucky enough to say something similar in a few years. We’re still too young to feel that much pain.

Posted: 2 August 2010 at 10:35 pm by Anthony

Couldn’t have put it better myself – this is pretty much our approach. It is working well enough that I don’t think we’ll ever have a dedicated support team, “You wrote it, you support it” seems to create just the right incentives. We seem to have a situation where a support incident almost always generates an internal Jira ticket to fix the problem at source, which is excellent.

One crucial point though… you have to let the developers have the time to fix the problems they are uncovering, or at least entertain a discussion about those fixes. Sometimes we jointly decide that it isn’t worth engineering ourselves out of a support issue, but everyone buys in to that. Most of the time they’ll put the fix in almost straight away, and we build slack into our development schedules to allow for both the time they spend on support issues and also the inevitable fixes.

I don’t think it will work if you have a very rigid planning system or little slack in the development process, but if you do it works a treat.

Posted: 4 August 2010 at 6:53 am by Simon Coles

Last time I reported a bug, I was told by an engineer that I should write a fix for it and submit a patch.

It’s the response I would expect from an open source project, but not a paid service.

lewcirne Reply:
August 4th, 2010 at 3:19 pm

Hi John,

I’m very sorry that you had this experience. If the issue you had was indeed a bug then we should take responsibility for fixing it. When we receive a feature request for our agent, or a request to support a new environment we don’t currently support, then we often encourage people to submit improvements to our community-contributed agent and we’ll make efforts to incorporate that goodness into our supported agent in due course.

If you feel that the issue is a legitimate bug, please forward any information you have about it to my attention and we’ll have a look at it. My email address is lew at (our domain name)

Posted: 4 August 2010 at 2:59 pm by John

Great article. You forgot to mention the free lunch plan for customers who all already got all NewRelic they can have.

Posted: 20 August 2010 at 10:36 am by rubyorchard

I’m curious, does your support staff have “write” access to production data? I know, unusual question, but I currently work in a culture where the support staff demands “write” access as the only way they know to support our programs and they have been doing it for years. I’ve been using the same argument that “developers like to code their way out of the support queue rather than execute manual repetitive tasks”. So let the developers support these requests and we’ll get rid of silly manual repetitive tasks and hopefully reduce the number of support requests/issues! thanks.

lewcirne Reply:
February 22nd, 2011 at 1:03 pm

Hi Cory,

In general, our philosophy is to empower the individual contributor as liberally as possible so that they can solve a customer’s issue as rapidly as possible using their own judgement. So yes, our support and development engineers have write access to the production environment to do things like update customer account status or turn on free gold access for a customer who has had a bad experience. We remind our employees of that old saying “with much power comes much responsibility” and trust them to do the right thing.

Posted: 22 February 2011 at 5:52 am by Cory

Blog

Tweets

Subscribe

How to Support 5,000 Customers with No Support Staff