This post was co-written by Bill Kayser and Joachim Meyer. Bill joined New Relic in 2008 and is now a Principal Architect working on the core application, alerting and availability monitoring. Joachim is a Software Engineer who works on all things business related in the core application.
One of the first things Ruby developers see when they use New Relic is the depth of insight into their web-based application, whether implemented with Rails or another framework. But for many customers, web applications are only part of the story in their production environments and monitoring background jobs is just as critical for operations as tracking what the users are experiencing. Two of the most common tools for running Ruby background jobs are Resque and DelayedJob. Fortunately, the New Relic Ruby Agent has built in support to monitor tasks executed by both of these. Resque and DelayedJob users can monitor the load and turnaround times for those jobs and browse errors and detailed traces on their execution without any additional special configuration.
But what if you run your background jobs with a rake task or some other job runner?
In our own application stack we use DelayedJob for sending email and other short asynchrnous tasks. But longer maintenance jobs are scheduled with cron using a regular job runner. We want the same visibility into these jobs we get with DelayedJob and ActionController so we just needed to write a bit of custom code in the script which we will share with you in this post.
The sample code in this post is available with a little more detail here.
Background Task Example
Let’s start with a typical example:
In order to instrument a job like this, there are a couple of things to accomplish. Let’s assume you’ve already set up New Relic for the app following the usual instructions for installing the Ruby Agent. In that case you need to:
* Tell the agent to start up and
* Tell the agent which transaction to record
Your Job class should look like this:
A couple of things to note:
* It’s necessary to specify the :category in the transaction_tracer in order to have the #run task appear under “Background Tasks” in the New Relic UI. Otherwise it will be treated like a web request.
* The :name option on add_transaction_tracer allows a custom label for the job when appearing in the list of Background Tasks in the UI.
* The NewRelic::Agent.manual_start method takes a hash of override values for agent configuration which is otherwise specified in newrelic.yml.
* This example overrides the transaction tracing threshold, or the minimum execution time required before a transaction trace is capture. In practice you might even want to disable transaction traces all together if your traced task takes a very long time to execute, as the memory overhead can increase the longer a transaction runs. We generally keep transaction tracing on for anything under an hour.
* Setting the app name in the code might not be ideal if you have staging environments. If setting the value in the newrelic.yml file isn’t convenient, you can also set it in the environment variable NEWRELIC_APP_NAME.
Background tasks instrumented this way will give you a nice presentation of how the execution time breaks down as well as a list of transaction traces for the task.
Adding Deeper Visibility
Generally we like to trace finer grained methods as tasks, like #process_account(acct) instead of the main #run loop. This gives us a little better granularity into things like the number of models being processed and the average time it takes to process each model.
Here’s how it may look for a job that processes accounts one at a time:
You can also add visibility into particular methods so you can track things not otherwise called out specifically, like a block of code that purges old data or updates a cache. These methods will then show up in the breakdown chart for the background task so you can see how much time is spent in them and how frequently they are invoked.
Expanding on the account processing example, here’s how we could get more visibility into the check_usage and send_invoice parts of the job:
Finally, there’s one more thing you can do to round out the data captured for background transactions. You can add custom key/value pairs that are captured and included in traced errors and transaction traces. This mechanism is available in web transactions as well and is usually implemented in a before filter in the application controller. In background jobs you would insert this line anywhere inside of the method being traced:
When viewing traced errors and transaction traces the custom parameters will appear alongside the trace.
These tips should help you get started monitoring your background jobs. In our next post on this topic, we’ll show you some strategies and code that will help you leverage New Relic for complete and consistent monitoring across all your background jobs.