Good news, PHP developers! If you use the PHPUnit library to manage and run your unit tests, you can now send test summary results to New Relic Insights and display useful information about test successes and failures.
Why this is useful
A dashboard of test summary data can allow you to quickly see a snapshot of overall test success as well as dig into failing suites to determine their cause. With a dashboard you can see at a glance that, for example, a particular test suite has suddenly started failing after that last commit. Clicking into that suite’s run ID lets you see that a particular test was the culprit because it failed to assert that a variable was true.
Tracking your test history also allows you to see patterns in the results. You can compare the number of failing tests this week to the number from last week. You can determine that a failing test only fails on a particular test machine, or perhaps it only fails 20% of the time. Perhaps you discover that one integration test normally completes quickly, except for 10% of the time when it takes forever.
New Relic is designed to make it easy to generate and meaningfully display your test events. Let’s walk through some examples and spark some ideas for your own test dashboard.
How to get started
This feature is disabled by default. To enable it, find or add newrelic.phpunit_events.enabled in your newrelic.ini file and set it to true. You’ll need to restart Apache or your alternate web dispatcher for the change to take effect.
Please note that exception messages are collected and sent with events. Additionally, if you use PHPUnit’s –disallow-test-output option, which flags tests that print output as “risky,” the test event will include the offending output.
When enabled, the PHP Agent detects PHPUnit commands and populates New Relic Insights with a set of attributes for the test suite and its individual test data. You can query the data with NRQL queries and set up a dashboard of information important to you.
The examples below illustrate the kinds of widgets you can create to answer questions and gain insights. For the NRQL query strings behind the widgets, check out our documentation page.
Percent success: What percent of suites or tests are passing?
Test outcome: What is the breakdown of test outcomes?
Test failures: What percent of the time does each test pass?
Test suite failures: What percent of the time does each suite pass, and is that consistent over time?
Duration: How long does each test suite take to run, and is that consistent over time?
Because a PHPUnit test suite is linked to individual tests via its run ID, you can use FACET widgets to filter results for a specific test run. For example, if you create a widget with the most recent failing test suites and link it to the current dashboard, you can click on a suite run ID and the surrounding widgets will update with information for just that suite.
In the example below, you can see that by clicking on run 77a075a14a078e81, which ran the suite FlakyTest, we learn that one of two tests, testFoo, failed.
To learn more about how to build and edit Insight dashboards, check out our awesome documentation.