The SaaS Litmus Test

I’ve read a lot of definitions of SaaS. Wikipedia has it covered of course. And there are countless sites describing how SaaS works and why it is valuable. I recommend you read those definitions and articles if you’re interested in learning more about SaaS and the underlying technology architecture, such as multitenancy.

That is not what this blog post is about.

Here at New Relic, we believe deeply that our customers gain significant benefits because we provide a 100% pure unadulterated SaaS solution. In fact, ensuring that we continue to build and run a pure SaaS business is a core tenet of New Relic — it’s ingrained in the DNA of our company. This means that every new feature, partnership and business practice must past the SaaS litmus test.

Therefore, the purpose of this post is to provide you with a SaaS buyers litmus test of sorts. The goal being to help you answer the question of whether your software vendor is providing you with a PURE or TRUE SaaS solution, or if you are being sold a SaaS(ish) solution that will leave you disappointed when the true benefits of SaaS are unachievable.

So, run through these tests when evaluating your SaaS solution. You could keep score (maybe a SaaS-o-meter?) to help you determine if the solution is really SaaS. Or you could keep it more basic and say if the vendor fails any of these tests they may be SaaS(ish), but they aren’t TRUE SaaS!

Test 1: Do I have to talk to a sales rep just to evaluate the solution?
If you’ve answered yes to this question, then subtract 1 from your SaaS-o-meter. One of the initial benefits of SaaS is ease of evaluation. You should be able to get started with a free trial of the solution on your own, without waiting on a sales representative to qualify you, setup your account, or do whatever “magic” is required to get your trial started.  If the solution is really SaaS, all you should need to do to is create an account on your vendor’s website and depending on the solution, you may need to install some software.

Test 2: Do you get automatic (frequent) updates and new features or do you get stuck with a specific version of the product?
If you’re not getting automatic updates, then subtract 1 from your SaaS-o-meter. A true SaaS solution maintains a single shared instance of the application. When the vendor makes updates to the solution or adds new features, all of their customers automatically benefit from these updates. This single instance model allows solution providers to be more agile in their product development cycles — providing updates and new capabilities at a far more rapid pace than their SaaS(ish) counterparts. So, if you’re waiting around for six months for your personal instance of the SaaS application to be upgraded with the latest and greatest features, then you really aren’t using SaaS — you’re essentially using a managed hosting solution.

Test 3: Can I buy the solution online?
If you answered no to this question, then subtract… OK — I think you get the point. If you’re evaluating the solution and your evaluation period is about to expire and you want to buy it, the solution should let you put in a credit card and subscribe to the service. If you are forced to call a sales rep to make the purchase, it isn’t SaaS. You may prefer to talk to a sales rep and not put the purchase on you credit card, but I guarantee any vendor will let you do that. Of course you may not want to keep it on your credit card indefinitely, but having that as an option lets you get rolling with the service fast, while you work out the longer-term payment options.

Test 4: Is there a free tier of the service that is also delivered via SaaS?
These days, any true SaaS solution has a Freemium business model. In short, you should be able to use a limited version of the product (yes, it’s limited because it’s free) forever on the same SaaS infrastructure as paying customers use. Why is this important? Well, someday you may want to upgrade from the free version to a paid version. And having your data live on through this transition is going to be critical.  You don’t want to be saying, “I’ve used your free service for a year. Now I’m ready to pay for your premium services and you’re telling me I have to start over. My historical data is gone!”

Test 5: Can you scale your use of the solution automatically?
This one is a little tricky because lots of vendors will say their solution allows this and it’s kind of hard to test until you actually need to do it. One way to gauge this is to use the initial setup / installation process as a proxy for how the scaling will go. But there are really two aspects to this test that you should consider before buying. First, does your vendor actually provide a flexible or streamlined sales process for scaling your usage beyond your current plan? If it takes weeks or months to add more users or another widget — well that’s just not SaaS. The other aspect is how easily the actual technology scales, which may not apply for all SaaS solutions (e.g. Salesforce.com). If there is a software component that needs to be installed (say an application monitoring agent), then verify that the installation process can be easily automated. You don’t want to find yourself manually installing software on 1,000 servers one at a time.

Test 6: Is the entire solution delivered via SaaS?
Some vendors are in a rush to have a SaaS offering and as it turns out building a real SaaS solution isn’t all that easy. To get their SaaS checkmark, many vendors put one or two features into a SaaS offering and leave the bulk of the functionality in their on-premise solution. You’ll either end up with some hybrid SaaS / on-premise solution to get the full feature set or end up with a SaaS solution that has a small subset of the features you actually want.

So, there you have it — six key tests you can use to determine if a solution is SaaS or SaaS(ish). Certainly, one of the benefits of SaaS is having the software and supporting infrastructure hosted and managed by a third party. But that’s just the basics. You’ll want more out of your SaaS solution and I think these six tests will help you make the right decision.

Plane Sky courtesy of Tony.Gonzalez, on Flickr

As Sr Director of Product Marketing at New Relic, Bill leads a stellar team of product marketing experts. Our primary goal in life is educating all our customers about the wonderful products and capabilities in the New Relic Software Analytics product suite. View posts by .

Interested in writing for New Relic Blog? Send us a pitch!