At New Relic, we believe that every one of our engineers is responsible for testing the quality of our product. In fact, we don’t even have a Quality Assurance team. As we do daily deploys to our site, we need a fast and reliable testing process to guarantee we are releasing the best product we can in a very short period of time. As our product’s and customers’ needs change, so do our testing methods. And we are constantly evaluating this process to find ways to improve upon it.
Life with ATDD
With quick release cycles, every link in our chain needs to be strong. We can’t afford to test the whole product right before we deploy. To manage this successfully, we have a process in place to help regulate the testing cycle. The Acceptance Test Driven Development (ATDD) cycle is a tried and true method developed by QA gurus meditating on a high mountain in the Himalayas. It helps regulate development milestones into bite sized chunks to assure our goals are met with success.
At New Relic, we do our best to follow the ATDD cycle. It makes developing our software more manageable and our lives a lot easier. Plus, everyone here loves demos : )
The 24 Hour QA Cycle
New Relic’s QA engineers are embedded throughout the organization. Each day, they work with our development, support, sales, and marketing teams to make sure we all stay focused on our goals. This ensures we have the same expectations on how a feature will work and what tests should be executed against it. A mix of unit, integration and functional tests are developed, maintained and updated as needed. And our testing framework is an equal miix of automated and manual testing.
Our offshore team is one of our secret weapons for QA. They work 12 hours ahead of us in the States and can execute a gauntlet of tests overnight. When we come to work bright and shiny in the morning, our inboxes are full of test suite reports, fixed issue validations and filed bug tickets. I like to think we have a 24 hour development cycle, which gives us the benefit of deploying often with success, low risk and an agile mindset.
Wingmen of the World Unite
At our Portland office, we have a prestigious title called the ‘Wingman’. The purpose of this position is to act as a second pair of eyes on any code before it’s ready for prime time. Responsibilites for this position include, but are not limited to, the following:
* Verifying the code works as described and that the change improves the general state of the code base.
* Making sure test scenarios exist for risky code changes.
* Checking for test coverage and legible code comments.
* Confirming any UI changes match our design playbook.
Our Secret Sauce
The final spicy special ingredient for the quality process involves using New Relic on New Relic with New Relic reporting about New Relic. We constanly monitor our own production and development environments with our product. We rely on this feedback for performance monitoring, code and database tuning, and UI workflow improvements, and this data is invaluable to us. The instant feedback lets us turnaround our code quickly, so we can check it in and deploy it faster.
The Big Picture
All these steps help define the lifecycle of our product, which in turn gives our customers new features and product improvements on a daily basis. Using New Relic to test, monitor and improve New Relic has proven to be our most important secret weapon.
In the words of New Relic Engineer Brent Miller, “We don’t eat our own dog food. We drink our own champagne.”