I’m very excited to introduce one of our coolest new features: Alert Policies! We heard loud and clear from our customers that they want more control over their alerting to reduce noise and increase signal.
With Alert Policies you can now manage all of your Applications, Key Transactions, Servers and Alert Channels from one unified dashboard. Alerting has never been so easy… Let’s take a closer look.
Alert Policies allow you to configure and share Alert Conditions and Alert Channels to many Servers, Applications and Key Transactions with similar alerting requirements. For example, you might have one Alert Policy for your production applications and another for development.
You can also create as many Alert Channels as you need using Email and our integrations with PagerDuty, HipChat, CampFire, and Webhooks. Taking it a step further, you can now combine any of your Alert Channels into Groups for easier setup, maintenance, and sharing across different Alert Policies.
We’ve been using these new alerting capabilities for the last few months (yes, we use New Relic to monitor New Relic) and it’s made alert management a breeze. They allow us to organize many of our Alert Policies and Groups to match the makeup of our teams and their areas of responsibility.
For example, our Site Engineering team has created an Alert Group that notifies their shared email address, HipChat room, and their PagerDuty escalation policy. This Alert Group is assigned to a Production Servers Policy so that they only get alerted for problems impacting production servers (i.e. they don’t want to get woken up at night because a development server is running low on memory).
Then they can create a separate Dev Servers Policy that only sends issues to low priority HipChat room.
Let’s take a look at how you can take advantage of these great new alert management capabilities in New Relic.
Starting today, new accounts will have access to these new alerting capabilities and existing accounts will be upgraded over the month of November. (Take a look at the Alert Policy FAQ for more details about the upgrade process.)
Once your account has Alert Policies enabled, login to New Relic and go to Tools > Alert Policies in the left navigation.
If you are an existing New Relic customer the upgrade process automatically creates policies for your Servers, Applications and Key transactions, based on your current alerting configuration.
You’ll likely want to modify your Alert Policies to better reflect the way you think about your infrastructure, applications and responsibilities.
We recommend the following workflow for optimizing your Alert Policies:
First, Add Your Alert Channels
Click on the ‘Channels and groups’ tab and scroll down to the Channels section.
New Relic supports PagerDuty, HipChat, CampFire, Webhooks and, of course, Email addresses. You can also send alerts to any user within your New Relic account, so you don’t need to add email addresses for them.
New Relic user accounts are used as alert channels and actually have some special controls to them that I’ll explain below. You can add more users from the ‘Account Settings’ menu item at the top right of your screen.
Add as many of these as you need by clicking the create channel button. Next you will be organizing them into reusable groups, so having these ready to go will make that step a lot easier.
Second, Create Alert Groups
Next we recommend creating Alert Groups for the various teams inside your business, or another logical grouping that fits the structure of your organization.
Alert Groups are great because if anything in your team changes, like adding a new team member or Alert Channel, you just make the change to the Alert Group and any policy that is using that Alert Group will automatically use the change.
Remember, Alert Groups are not only for grouping users. You can also mix in any other types of channels you want.
If you want to get really fancy you can create groups of groups as well. Configuration management will be more complex if you choose to do this, so I would only recommend it if you really think it will help.
Third, Create/Edit Your Alert Policies
There are a few things to consider before you start creating/editing your Alert Policies:
- All policy members (apps, key transactions, servers) share the same alert conditions
- All Alert Channels are alerted when a critical (red) threshold is crossed for any of the members
Here are a few questions we asked ourselves when we setup our own alert policies:
- Is this app, key transaction or server part of our development, staging, or production environment?
- Does this app, key transaction or server need it’s own ‘special snowflake’ set of alert conditions or channels? If so, it will need its own policy.
After you’ve considered these you can start crafting your policies and at this point it’s four simple steps. Here is an example for creating an Alert Policy for your production servers:
- In the Server policy tab and create a new policy named Production Servers.
- Set your condition thresholds
- Add your servers
- Add your notification channels
Are there any other new features that come with Alert Policies?
With the Alert Policy release our main goal was to reduce noise and increase signal by ensuring the right people were receiving alerts for the things they care about.
However we were able to sneak in a couple other features as well.
Server policies have Downtime alerting! When enabled, alerts will be triggered when New Relic stops receiving metrics from that server via the server monitoring agent for the specified duration.
Apdex alert conditions for Applications and Key Transactions now have a user configurable duration setting allowing you to smooth out your alerts.