Automated Testing at the Functional Layer

Those of you that have read my previous posts have probably noticed a trend. Automation is something I am very passionate about. We’ve talked about automating the release process, automating the little things, but haven’t talked about one of the more important areas of automation in the quality space: automated functional testing.

What are they?

Functional tests make up yet another layer of the quality cake. Not enough on their own, they are most effective when complimented with layers of unit and integration tests. Typically the most involved layer of testing, they often represent end-to-end tests that execute against live web sites, installed client software, or the like. They are also the closest thing to simulating a user interacting with the software. Having been part of both web and client software shops, I have experienced struggles and victories in both arenas.

Anyone who writes software knows how easy it is to make changes in one area that result in unintended side effects result in another, seemingly unrelated area. Due to their end-to-end nature, many things may trigger a failure. This can be both a blessing and curse — a blessing in that a functional test may detect a failure in an area other than the segment under test, and a curse for the exact same reason, in that it has the potential to obfuscate the debugging process. In many cases, when a test has failed as a result of a change that is not a bug per se, it often serves as a catalyst for healthy conversation resulting in additional refactoring, development, or future enhancements. We on the Windows team get the most value out of our functional suite when making big changes like merging branches to mainline or refactoring large sections of code.

Nuts and bolts

There are a number of ways to implement a suite, and a multitude of both commercial and open-source tools available. For teams lacking experienced engineers, an out-of-the-box commercial solution may be more appropriate. For others, it can be simple enough to build a class or function library on top of one of the many unit testing frameworks available. Both at New Relic and elsewhere I have had success using the latter approach where a basic test involves invoking an action (starting a process or making a web request against a test site), then using a basic unit testing framework to assert an element of data. Depending on the area under test, this ‘actual value’ can consist of data from your application, data in a log file, or even system-level data such as files or registry keys.

Just as the 1’s and 0’s are important, the approach is equally important. Naturally, you may be tempted to say “every code change needs to be backed by a functional test”. Approach each new feature or bug fix as a talking point in determining where appropriate test coverage makes sense. Make this conversation part of your quality/testing/release plan. Depending on the data points available in your application, it may not make sense for every feature or change to be backed by a functional test. Like all other quality gates, your functional tests should be configured to run as part of your CI pipeline. When it comes to source control, we’ve had great success storing our functional tests in the same repository as the code under test, but in a different project. With this approach, tests can be created and executed in a development branch, then merged to the main branch along with the code changes when the feature is complete.

So what’s the catch?

Functional tests are the most expensive to implement and maintain, but are quite valuable, as they test a layer that is hard, if not impossible, for unit and integration tests. Out of the many layers of tests, functional tests are some of the most sensitive; it may take some creativeness to get all of your tests to pass as consistently as unit tests. They are best implemented by QA developers, due to their code-heavy nature. It’s crucial to treat your functional tests like you would the rest of your code by following good software development principles. Despite their added costs, they typically serve as the last line of defense before code is delivered to the customer.

The final word

Here’s the bottom line: Functional tests are an important and powerful piece of the CI process. However, don’t put all of your eggs in one basket. A high-functioning CI process takes a layered approach to testing, and your functional tests are just another layer. Don’t be discouraged by the challenges you may face. Trust me, you’ll be glad you made the effort the first time you find yourself saying “wow, I’m glad that bug didn’t get out!”. Have fun!!

Matt Sneeden is a QA Developer at New Relic. He supports the .NET Agent team, specializing in building continuous integration and automated testing systems. When not wired in, he can be found exploring the outdoors of southern Wisconsin. View posts by .

Interested in writing for New Relic Blog? Send us a pitch!