Customizing How Transactions are Monitored in Python

When monitoring any Python web application in New Relic, you may come across specific web transactions that don’t fall within the normal range of what would be considered reasonable for your web application. Sometimes this could be a temporary situation which arises when you deploy some code that breaks your application. There also may be a specific web transaction you know is suboptimal, but addressing the underlying cause isn’t a high priority.

One example of such an abnormal situation is a very long running request skewing the average response time and Apdex score on the overview dashboard. Another is where a web transaction may be experiencing a high error rate, dragging down the Apdex score for your application.

In either situation, the out-of-band results can have a skewing effect on summary data presented for your application, making it appear that your application is performing worse than it is. Performance issues are not something you should generally ignore, but in some cases it can be necessary to do just that. To do so while getting an accurate view of your performance elsewhere, you can change how the web transactions are monitored or categorized.

In this post I’ll describe how you accomplish this through the New Relic UI by adjusting threshold values used in Apdex calculations for web transactions. I’ll also cover how you can filter out or otherwise change how specific web transactions are monitored using our agent API in the Python agent.

Measuring user satisfaction

First, I’ll give you more insight into New Relic’s Apdex score, how it’s determined, and what it represents. Apdex is ultimately a metric for determining user satisfaction. The calculation for the Apdex score takes into consideration the response time for a web transaction, as well as whether any errors occurred in handling it.

The level of user satisfaction for a specific web transaction will be classified as being satisfied, tolerating or frustrated. For a successful web transaction, the level of user satisfaction depends on the response time, and can be any of those three. In the event of an error, the user is immediately seen as being frustrated. Your overall Apdex score will aggregate the results for each distinct web transaction across your entire web application.

Here’s a look at how exactly this is calculated:

Apdex Calculations

This is then presented as a single Apdex score, along with a historical chart on the overview dashboard in the New Relic UI. The purpose of the score is to give a single, easily understood value which gives a sense of the overall health of your application.

By default, the threshold value used to calculate the Apdex score is 0.5. This means that if a web transaction takes more that 0.5 seconds, the user satisfaction will be seen as tolerating, rather than satisfied. If it takes more than 2.0 seconds, the user satisfaction will be seen as frustrated.

Now, although the response times for web transactions across your web application may be consistent, this default value may not be suitable at the outset. One of the first things you should do is to adjust the threshold value used in this calculation, increasing it to a value which is more in line with the general performance of your application. As you then address any performance bottlenecks in your application and reduce response times, you can gradually drop the threshold value again to what you see as acceptable performance.

The Apdex score is therefore not an absolute measure, but a value you can adjust and use as your own personal target for driving improvement. You can find more information on the Apdex score and how it is used in our knowledge base.

Impact of key transactions on Apdex

In practice, web transaction response times across the whole of your web application are not going to be consistent. Instead you’ll have a variety where some exhibit quick response times, and others exhibit much longer response times.

The problem with this is that the default value used to calculate the Apdex score won’t be appropriate for all web transactions. By relying on the default value, a specific web transaction that consistently exhibits long response times would always record the user satisfaction as being frustrated, when in practice such a long response time is expected (such as uploading files to a web application).

At the other end of the spectrum, certain high throughput web transactions will need to have as short a response time as possible. In this case the default value used to calculate the Apdex score may indicate that a user is always satisfied, when in fact they may not be, due to the importance of that web transaction and the need for a quick response.

To address these disparities and the need to treat certain web transactions differently, what you can do is set those web transactions with behaviour outside of the normal range as key transactions.

When setting up key transactions, the threshold value used in the Apdex score calculation can be individually set for that one web transaction. That way, you can accommodate the distinct behaviour and have them contribute in a more realistic way to the overall Apdex score for the whole web application.

A separate chart for the Apdex score is also available on the key transactions page for the web transaction, allowing you to then monitor that specific web transaction in isolation from the web application as a whole.

Excluding transactions from Apdex

Although key transactions provide a way of adjusting the threshold value used in Apdex score calculations for individual web transactions, certain types of web transactions can still present problems.

The classic example of this is where a web transaction is handling uploading files to the web application. Here the file sizes can vary greatly and the speed of a client network can be an issue, so it would never be easy to calculate an appropriate threshold value for the Apdex calculation.

In this case it makes more sense to be able to exclude specific web transactions from the Apdex score calculation. To do this it’s necessary to add a call to our agent API into the code of the request handler of your web application for the relevant web transaction:


import newrelic.agent
newrelic.agent.suppress_apdex_metric()

If using Apache/mod_wsgi, a second option is to use the SetEnv directive within the Apache configuration file to set an option for the URLs corresponding to the target web transaction. Restricting it to the specific URLs is performed by using the Location directive in the Apache configuration.


<Location /admin/upload>
SetEnv newrelic.suppress_apdex_metric true
</Location>

The value of the setting can be ‘on‘, ‘true‘ or ‘1‘.

The SetEnv directive will set the specified key/value in the WSGI environ dictionary passed through to the WSGI application for each request. Our instrumentation wrapper for the WSGI application will automatically detect the setting and subsequently suppress the generation of the Apdex metric for that web transaction.

Marking transactions as complete

Suppressing the generation of the Apdex metric for a web transaction will ensure that the overall Apdex score is not unduly affected, but in the case of long response times, it can still skew the average response time given for the whole web application on the overview dashboard.

To combat this, the next step you can take is to prematurely mark the web transaction as complete after the initial processing of the web request details, and before any potentially time consuming operation which leads to the extended response time.

If this approach seems more appropriate, you can once again make a call to our agent API in the code of the request handler.


import newrelic.agent
newrelic.agent.end_of_transaction()

There’s one thing to be careful of when using this approach: because the web transaction is deemed as complete at that point, yet the request thread is still tied up in completing the request, it can upset calculations used to produce the capacity analysis report. That is, the lack of monitoring for the remainder of the web transaction can result in the capacity analysis report indicating that the server is utilizing less capacity than it actually is.

A further issue is that any error that is raised during the remainder of the web transaction will not be recorded.

Ignoring the transaction completely

An even more drastic action you can take is to simply ignore a specific web transaction completely. The agent API call in this case is:


import newrelic.agent
newrelic.agent.ignore_transaction()

If using Apache/mod_wsgi, you could indicate that a web transaction should be ignored completely by using:


<Location /admin/upload>
SetEnv newrelic.ignore_transaction true
</Location>

In ignoring the web transaction, any errors which occur will not be recorded, as with marking a transaction as complete early. Ignoring transactions can similarly lead to a misleading capacity analysis report.

Recording as a background task

As I’ve mentioned, marking transactions as complete early on, or ignoring the web transaction completely, will result in some loss of information and visibility into what is going on in your application. A better alternative may be to indicate that the web transaction should actually be flagged as a background task:


import newrelic.agent
newrelic.agent.set_background_task()

If using Apache/mod_wsgi, you can indicate that a web transaction should be recorded as a background task by using:


<Location /admin/upload>
SetEnv newrelic.set_background_task true
</Location>

By marking a web transaction as a background task, it will be reported distinctly from the normal web transactions in the New Relic UI. It will also no longer contribute to the Apdex score. All the normal methods for viewing the breakdown of where time was spent in handling the web request will be available, as will recording of errors and slow transaction samples.

Reporting to distinct applications

One final approach that can be taken if you are using Apache/mod_wsgi is to logically break up your application so that data from different parts of the web application report to distinct application entries within the New Relic UI.

This particular approach is very useful if you’re using the Django web framework and want to differentiate the Django admin interface so it reports separately. This can help because the web transactions handled by the admin interface quite often perform actions that take a lot longer than your typical web transactions. It is therefore advantageous to view them separately from the rest of the web application.

This can be done by once again using SetEnv in the Apache configuration file. This time we override the application name which the data should be reported to for any web transaction falling under the ‘/admin‘ URL.


<Location /admin>
SetEnv newrelic.app_name 'Python Application (Admin)'
</Location>

Note that although it is possible to have a web application report to multiple applications in the New Relic UI in this way, we would generally recommend that data not be reported to more that 3 distinct applications within one web application process. Reporting to too many distinct applications could impact the performance of the web application.

Not all web transactions are equal

The lesson to be learned here is that not all web transactions handled by a web application will behave in a similar way. Features like key transactions can be used to override how specific web transactions are monitored, allowing you to treat them differently from the rest of the web app. It can also be advantageous to logically partition your web application by viewing it as actually being composed of separate applications. If necessary, you can also drop down to embedding calls to the agent API to further override how the agent treats specific web transactions.

The most appropriate technique for you will ultimately depend on what your specific web application is doing. Hopefully I’ve succeeded in outlining the different choices you have and given you a head start on making use of the agent’s capabilities so you can get better, actionable data out of your web application.

Graham has been doing programming for over 25 years. He is the author of mod_wsgi, the leading Python web hosting solution for Apache and is a member of both the Python and Apache Software Foundations. He works on bringing New Relic to the Python world. View posts by .

Interested in writing for New Relic Blog? Send us a pitch!