Monitoring Django management commands

Our prime focus at New Relic has traditionally been monitoring of web applications. However, most sites will be comprised of a lot more than just the web app — there can also be persistent backend services for executing background tasks, or even individual scripts that are periodically run from scheduling systems such as UNIX cron. As a result, our different language agents also provide various abilities to monitor activity occurring in such additional applications, reporting the data collected as background tasks.

In the case of the Python agent, we provide built-in support for the Celery and gearman task queueing systems and allow the execution of tasks to be automatically monitored. Beyond those two specific systems, if you want to monitor background tasks in Python, you'll need to start using our agent APIs to instrument the application you wish to have monitored. For simple cases the need to intrusively instrument an application for monitoring can be avoided by setting up instrumentation via the agent configuration file.

In this post we'll describe how you can use the agent configuration file to set up monitoring of background tasks by using as an example Django management commands executed from the 'manage.py' script created when you first started your Django project.

Defining the function to be instrumented

To define the function within the Python application to be instrumented as the entry point for a background task, a 'background-task' section should be added to the agent configuration file with name of the form 'background-task:name'. The 'name' part can be anything so long as each 'background-task' section in the agent configuration file has a unique name.

In the case of wanting to monitor Django management commands, if we wish to monitor the 'syncdb' command we need to identify the function which implements that command in the Django code base. The fully qualified name for that function in the form 'module:function' should then be listed under the 'function' setting within the 'background-task' section you have created. The 'enabled' setting should also be set to 'true' to have the agent use this 'background-task' section.

[background-task:django-manage-syncdb]

enabled = true

function = django.core.management.commands.syncdb:Command.handle_noargs

name = syncdb

group = Django

If we were to only set 'enabled' and 'function' then when the calls to that function are reported, they'd be reported with a transaction name including the fully qualified name of the function. It would appear under the 'Function' group for 'Other transactions' in the 'Transactions' tab of the New Relic UI for the application the data is reported to.

If we wish to override the name and the group under which it's reported, we can set the 'name' and 'group' settings as well. The 'group' should be a single name without any '/' appearing in it. The 'name' can be any value you want. In this case we are setting it to just be 'syncdb', which is the name of the sub command you would give to the Django 'manage.py' script when run. With the values shown, this will now be reported under the 'Django' group for 'Other transactions' in the 'Transactions' tab, with the name 'syncdb'.

Ensuring that the agent has registered itself

Normally when the Python agent is used to monitor a web application, the agent will only register itself lazily within a parallel background thread when the first web request is received by the web application being monitored. Until registration of the agent completes and server side configuration is received, the web requests will not actually be monitored. This means that there is a small window of time where data will not be collected for the web requests as everything gets set up.

In the case of wanting to monitor a Django management command, having the agent register itself lazily in this way would mean that we'd completely miss monitoring of the execution of the Django management command since the function to be monitored will only be called the once.

What we therefore need to do is configure the agent to not perform registration in the background, but to perform it immediately and wait for registration to occur the first time before actually proceeding with the execution of the function to be monitored.

To do this we need to set the 'startup_timeout' setting in the 'newrelic' section of the agent configuration file. In the case where the same agent configuration file is being used when monitoring a separate web application, setting the 'startup_timeout' setting in the default 'newrelic' section should actually be avoided, as it would consequently cause the serving of web requests to be stalled as the agent registered itself. For a web application having startup delayed in this way would generally not be desirable.

What you should do instead is create a separate environment section in the agent configuration file. Instead of just 'newrelic', the name of this section should use the form 'newrelic:name' where 'name' is the name to be given to that specific environment configuration. For this case we will call the environment 'django-manage'.

[newrelic:django-manage]

app_name = Python Application (Django Management Commands)

startup_timeout = 10.0

By using a separate environment section, we can also separately set the 'app_name' setting so that data for the management commands are reported into a separate application in the New Relic UI for any web application being monitored. This is recommended — otherwise charts for CPU and memory usage, as well as reports like the capacity analysis, will become confusing due to the mix of application types being reported together.

Running the Django management command

To now run the Django management command and have it be monitored, we need to set both the 'NEW_RELIC_CONFIG_FILE' environment variable to the path to the agent configuration file, but also set the 'NEW_RELIC_ENVIRONMENT' environment variable to the name of the environment section if used. The 'newrelic-admin' wrapper script is then used with the 'run-python' option to run the 'manage.py' script, replacing the use of the 'python' program. If executing the command from within a shell environment, you would now use:

NEW_RELIC_CONFIG_FILE=newrelic.ini

NEW_RELIC_ENVIRONMENT=django-manage

export NEW_RELIC_CONFIG_FILE

export NEW_RELIC_ENVIRONMENT

newrelic-admin run-python manage.py syncdb

When run, if for some reason agent registration takes too long, the agent will timeout after the specified time and the management command will be allowed to proceed anyway. Whether the agent registration is quick or not, the need to wait for agent registration means that there will be a delay in actually starting the management command. You should keep this in mind if the management command needs to be run as quickly as possible. Where it is a long running command though, this is generally not an issue.

Do note though that this should not be used on a management command that enters into some type of main loop, as is the case with the 'runserver' command. Trying to monitor the 'runserver' command in particular would also interfere with normal monitoring of the web application which is started.

If when running the Django management command you find that data is not always being reliably uploaded on completion of the command due to network delays, a 'shutdown_timeout' setting can also be specified in the 'newrelic' environment section of the agent configuration. This will override the default timeout of 2.5 seconds for uploading data on process shutdown. The default is normally 2.5 seconds on shutdown when monitoring a web application, because web servers such as Apache when using mod_wsgi enforce a fixed 3.0 second timeout before the worker process would be forcibly killed.

The agent needs to timeout before then to allow other code time to run on shutdown. In the case of monitoring a one-off script, as in this case, the shutdown timeout can be increased as necessary, as there will be no process manager waiting to forcibly kill the process if it takes too long to complete uploading of all data.

What you should expect to see in New Relic

As described above, the monitored Django management command will be reported under the 'Django' group for 'Other transactions' in the 'Transactions' tab. The transaction names as listed will then correspond to whatever management commands you set up to be instrumented via the agent configuration file.

Built-in support for Django management commands

Although we have outlined here how you can set up instrumentation for monitoring specific Django management commands using the agent configuration file, we are also working on actual built-in support within the agent. At this time we have a prototype of the required instrumentation and are working through validating that the instrumentation of the Django management commands will not interfere with monitoring of a web application when the WSGI server is started up using a Django management command such as 'runserver', 'runfcgi' or 'run_gunicorn'. We hope to have the built-in support for monitoring the Django management commands available in a release of the Python agent not too far down the track.