This is a guest post written by Ariel Smoliar, Senior Product Manager at Sumo Logic, a cloud-based log management and analytics service that leverages Big Data to deliver real-time IT insights.
At New Relic’s recent user conference, FutureStack13, I had the opportunity to talk with New Relic users about software analytics to better understand the importance of the APM SaaS solution. Many of the users were looking for a complementary log management platform to their existing APM solution. A month later, at AWS re:Invent, I spoke with Sumo Logic users who were also seeking an APM solution to monitor the availability and performance of their applications.
As we were synthesizing these discussions, the main goal in front of us was how to best demonstrate both the Sumo Logic log management service and New Relic’s APM capabilities across multiple environments. The first step toward this target was to empower the integration for joint users of Sumo Logic and New Relic. We created a new plugin (following Sumo’s existing Windows Events plugin) that enables users to see their log events from the Sumo Log Analysis QuickStart application in their New Relic dashboards. In order to see the log events in New Relic, users need both Sumo Logic and New Relic accounts.
In order to bring the users of this integration up to speed with the capabilities of each solution, we decided to focus on the following use cases:
- Visitor Monitoring. E-commerce companies, for example, can obtain important information about the visitors on their website/application, such as email, frequent IPs and successful and failed logins. It will enable the DevOps team to quickly address customer-facing issues.
- Operations Monitoring. Infrastructure teams can track the patterns of important keywords (error, failure, timeout) across all their logs (web-serves, databases, etc.) that indicate issues or incidents, and react in real-time to prevent a potential outage or any other impact that can affect the company.
We hope that with this SaaS log management-APM integration solution new users can begin to gain deeper analytics and insights from their log events, independent of where they were generated, and utilize New Relic’s plugins platform. We will be adding more functionality to help our users to acquire additional security, operations and business intelligence insights.
Plugin Design and Data Visualization
Sumo Logic implemented a new Dashboard API to be able to send data from the Sumo Logic service to New Relic dashboards based on the monitor values. Once the Log Analysis Quickstart application dashboards are loaded and the New Relic plugin is installed, data from the monitors in the Visits dashboard and the Keywords and Metadata dashboard is sent to New Relic. From the Visits dashboard, data about frequent IP addresses, frequent email addresses, successful and failed logins and sessions (bar charts will appear in table format in New Relic Visits dashboard) is sent through the API.
From the Keywords and Metadata dashboard, one can find data around high/medium priority keywords and issues by origin/category/host sent through the API. Category and Host are searchable metadata tags for the messages that are attached to your log messages at collection time. These tags provide valuable keywords and terms to find targeted results in search queries.
Now New Relic users can see their log events appear in tables and continuous graphs in New Relic beside other data from their monitoring plugins:
Setting Alerts in Both Environments
Users can design and set alerts in both environments. In Sumo Logic, a user can set up an alert if the number of failed logins on your website exceeds the threshold. The method of setting an alert is by using a scheduled search and the user will be notified only when certain conditions arise.
In the design of the New Relic plugin, we defined three summary metrics: Error Keywords, Failed Logins and Session Errors.
The metrics values are also shown in the bottom of every dashboard:
Users of the plugin can set the alerts conditions for the metrics and receive notifications according to their configuration. The status updates appear under recent events.
One of the ways that New Relic sends alerts is through Webhook. The alerts can be ingested in Sumo Logic service using the HTTP source, but I will keep you curious for now as we are still learning the best way to visualize the data for our mutual users.
Nate Silver says that the signal is the truth and the noise is what distracts us from the truth – that is our biggest challenge.
We look forward to hearing from you where we can take this integration next and provide you even more effective real-time signals and analytics across your multiple environments.