In times of planned system disruptions, a steady stream of noisy, unnecessary alerts can be a major distraction.
That’s why we’re excited to tell you that New Relic Alerts now supports muting rules, enabling you to silence notifications during maintenance windows.
Muting rules make it possible to override alert policies and conditions you’ve defined in New Relic, so you can take more control over your alerts and suppress notifications during times of known system disruptions, such as maintenance windows, deployments, and during testing.
A muting rule in New Relic contains a set of conditions that match against a large number of attributes and tags present in a violation event, including:
- Alert policy and condition names
- Application and service names
- Host names
- AWS tags
- Kubernetes tags
- Any other tags associated with your entities or open telemetry
Muting rules tell New Relic how to identify individual violations after they are created, but before an incident is opened. They then override the default lifecycle of an alert to indicate that it should not send notifications.
During maintenance windows and system deployments, it is important to find a balance between filtering out the noise, while still maintaining observability and alerting on the state of the rest of your system.
By creating rules that identify violations—and not simply enabling or disabling alert conditions and monitors—you can target the specific subsets of your system that are being affected, and minimize risks associated with overmuting.
Additionally, by keeping alert conditions running but silent during these periods, you can easily observe when entities have recovered from maintenance, and have returned to a healthy state.
How do muting rules work?
With muting rules, your normal alert incident lifecycle is maintained; the only change is whether or not an incident triggers a notification. Violations and incidents are still opened, and health indicators will show the current state of your entities as you progress through your maintenance window.
When the conditions defined in a muting rule match on a violation event, the violation is marked as “muted.” When a violation is muted, the incident lifecycle is modified in the following ways (you can find a detailed workflow chart in the docs):
- When a muted violation opens an incident, an “open incident” notification will not be sent.
- If any other violation that is not muted is associated with that incident, then an “open incident” notification will be sent.
- If an incident sends an “open incident” notification, then all subsequent events will trigger notifications.
- If an incident never sent an “open incident” notification, then no subsequent events will trigger notifications.
Enterprise scale issue detection and incident management requires flexibility and automation. Muting notifications is the first in a series of lifecycle override actions that we will be releasing over the course of this year.
The next action to be released will allow you to disable incident creation for targeted violations. All of these controls will be manageable through our API and UI.
For this release, you can enable and disable muting rules using the NerdGraph API—New Relic’s GraphQL API. In addition, UI indicators in the New Relic Alerts page will show you violations and incidents that have been muted. (A UI for managing muting rules in New Relic One will be released very soon.)
Within the NerdGraph API, you set a name and description for your rule in addition to conditions, which describe the criteria used to match against attributes of a violation object. Sets of matching conditions can be combined to create complex filters. Filters can either use “or” or “and” to combine them, and most attribute evaluations can accept an array of possible values. You can find multiple NerdGraph APIs in the documentation.
How to get started
The ability to mute notifications is one way we want to give you more control over your monitoring and reduce alert fatigue.
Muting rules are now available to all New Relic customers with a pro subscription. Check out the muting rules documentation to begin using it in your account.
And if you’re not yet using New Relic and want to give it a try, you can sign up for a trial and get started for free.