New Relic customers often ask how to display open Alert incidents or violations on a New Relic dashboard. One approach is to configure New Relic Alerts to send incident data to New Relic Insights using a webhook, but that creates an event whenever an incident or violation is opened, acknowledged, or closed.

While this data is great for analyzing alert incidents after the fact—answering questions like “Which alert conditions are most frequently violated?” “Which entities generate the most violations?” and “What is the mean time to resolution (MTTR)?”—it doesn’t help if you want to see only incidents and violations that are currently open. You can view a list of open incidents and violations in the Alerts UI, but many customers would like to display this data on a dashboard, alongside other performance metrics.

Fortunately, the New Relic REST API includes endpoints that return lists of Alerts incidents and violations. These endpoints accept an only_open parameter to specify that they should return only items that are currently open. That’s great, but how do you display this information on a dashboard?

Follow along with this example to see how to create a dashboard showing open Alerts violations in two simple steps using a custom script.

1. Run the open violations script

The script is actually a New Relic Synthetics API test that takes advantage of Synthetics’ secure credentials. To use the script, you’ll need:

  1. Secure credentials for your New Relic REST API key, an Insights insert key, and your New Relic account ID.
  2. An API test monitor with the script shown below.
    Note: You’ll want to schedule the monitor to run from at least one location once per minute to minimize latency.

This script queries the New Relic REST API to get a list of open violations, iterates over that list, and generates an Insights custom event called AlertViolationsSample with information about each violation.

To use the script, paste it into the Synthetics script editor.

const request = require('request');

var headers = {
  'Content-Type': 'json/application',
  'X-Api-Key': $secure.API_KEY
};

var options = {
  url: 'https://api.newrelic.com/v2/alerts_violations.json',
  qs: { 'only_open': 'true' },
  headers: headers
};
 
var violations = null// Get open violations
request.get(options,
  function(err, response, body) {
    if (response.statusCode == 200) {
      var data = JSON.parse(body);
     violations = data.violations;
    // For each violation...
    for (var i = 0; i < violations.length; i++) {
      // ...create Insights event
     insertViolationEvent(violations[i]);
     }
   }
 }
);

function insertViolationEvent(violation) {
  var headers = {
    'Content-Type': 'json/application',
    'X-Insert-Key': $secure.INSERT_KEY
  };
  var options = {
    url: 'https://insights-collector.newrelic.com/v1/accounts/' + $secure.ACCOUNT_ID + '/events',
    headers: headers
  }
  options['body'] = JSON.stringify(
    {
      'eventType': 'AlertViolationsSample',
      'id': violation.id,
      'label': violation.label,
      'duration': violation.duration,
      'policy_name': violation.policy_name,
      'condition_name': violation.condition_name,
      'priority': violation.priority,
      'opened_at': violation.opened_at,
      'entity.product': violation.entity.product,
      'entity.type': violation.entity.type,
      'entity.id': violation.entity.id,
      'entity.name': violation.entity.name
    }
   );
   request.post(options);
}

2. Create the dashboard

Once you’ve gathered the data, you can use New Relic Insights or the New Relic One chart builder to query it using the New Relic Query Language (NRQL) and create a dashboard, as shown here:

SELECT
  latest(duration) / 60 AS 'Duration (mins)',
  latest(condition_name), latest(label), latest(policy_name),
  latest(`entity.name`), latest(priority)
FROM AlertViolationsSample
FACET id

A New Relic One dashboard that shows open violations in New Relic Alerts.

Enhancing the script as needed

If you want to run the script elsewhere, say as an AWS Lambda function, you just need to replace the secure credential values in the script (beginning with $secure) with your New Relic account values.

Also note that this script queries open violations from the REST API; you may prefer to see open incidents instead. Again, no problem: Just change the API endpoint in the script to https://api.newrelic.com/v2/alerts_incidents.json, and modify the values as necessary.

Finally, if you have a large number of open incidents or violations, the REST API paginates the results. You’ll have to examine the response and see if there are additional pages; if there are, you can call the API again, passing a page parameter to request the desired page.

If you’re interested in learning more about our APIs, check out the New Relic Developers Hub, and look for other examples in the New Relic Explorers Hub.

Phil Weber is a Senior Technical Training Specialist with New Relic University. He worked as a software developer for over 15 years, and has been a technical trainer since 2005. As a consultant, he worked for such clients as Intel and Kaiser Permanente. View posts by .

Interested in writing for New Relic Blog? Send us a pitch!