On the New Relic Security Team we track many events to monitor the security of our company and our users’ data. Conveniently, NRDB—the technology powering New Relic Insights—is an incredibly powerful event database.
To demonstrate just how powerful, “All Quiet on the Cyber Front” is a new series of fictionalized blog posts about how we use Insights to help us detect and respond to security threats. Names, dates, and facts have been changed to protect the innocent … and the not so innocent.
Episode 1: The Phantom Login
On the wall above the security team’s area in our Portland engineering offices, there sits a display home to various dashboards, many of which are fed by inserting custom events via the Insights API. Custom events let us track data not included in New Relic’s out-of-the-box settings, so we can tackle a wide range of security use cases. One dashboard that permanently adorns this display highlights which employees are logging in, from where, and at what times.
Our authentication dashboard was created by parsing logs from our SSO provider, and using them to create a custom event type called AuthEvents. This event type has several attributes, including the employee username, the country where the login request originated, and a message that indicates if the login was a success or failure.
We can then craft queries to keep track of large numbers of failed logins:
To detect when employees unexpectedly log in from outside the United States:
And to generate a list of the countries from where our employees are logging in:
One fine day, we noticed something on the dashboard that tingled our cyber-senses:
Hmm … Cobra Island? We definitely don’t have an office there, and no one mentioned any vacation plans. Let’s investigate by adding a filter!
With the filter added, we can see that Arthur Curry is either signing in while enjoying impromptu travel, or something possibly more nefarious is afoot. But wait! We use multifactor authentication company-wide to help protect employee accounts, and we send that data to Insights as another custom event type. Let’s run a New Relic Query Language (NRQL) query to investigate further!
The good news: It doesn’t look like anyone was actually able to access the account from Cobra Island.
The bad news: This is looking even more suspicious.
What next? Arthur does travel occasionally (mostly to Atlantis), so we should check and see if that’s the case. However, if his account is compromised, we can’t trust anything it might say. So let’s ping his manager and see if she can shed any light.
Thanks to a combination of New Relic Insights, our custom event types (pulled from our SSO provider and inserted via the Insights API), and good old-fashioned detective work, we know that Arthur was the victim of a phishing attack. Now we can work to identify where the attack came from, determine if any other employees were contacted, and block the fraudster from contacting additional employees. By the time Arthur has finished his refresher course on security awareness, we can be reasonably sure that this incident is at an end. (But, just in case, we’ll keep one eye on the dashboard.)