Custom events are a developer's best friend

If you've been following this series, you know how enthusiastic we, at New Relic, are about using New Relic ourselves to understand and improve our systems. We make extensive use internally of our platform and its capabilities just like our customers.

That said, there are some differences between our own use of New Relic and our typical customers. One of those differences is how much we use custom events. On average, our customers rely on 95% "out-of-the-box" instrumentation and 5% custom instrumentation. But within New Relic, over 30% of the telemetry data we collect about our systems is custom events.

What is a custom event?

It's whatever you want it to be!

You can name a custom event anything you want (well, almost anything), and store whatever attributes and values in it that you want. It's a lot like creating a new database table in your application model, except that you don't have to do anything to register it in advance. The unique schemaless nature of the New Relic database (NRDB) means you can decide to start sending a new custom event to New Relic any time you want, and it'll immediately be available for you to start querying.

A custom event must have an event type, analogous to the name of a database table, and you can optionally give it a timestamp. (If you don't provide a timestamp, we’ll automatically add one based on the time the event is received by New Relic.) Beyond that, you can add whatever additional field you want to record and describe the data point. If you decide later that you want to add or modify fields on the event, you can also do that with no need to tell NRDB in advance. Your new fields will be automatically detected and available in your queries.

Why are custom events awesome?

Out-of-the-box instrumentation is great, but as smart as they are, our agent engineers aren't mind readers. They make sure that as many common use cases as possible are instrumented, but they aren't the experts on your application and your business—you are!

At New Relic, we built an awesome massively-scalable distributed multi-tenant database. At the end of the day, all the data gets stored in files on disks somewhere, but a lot can happen during the lifetime of an "archive file," so we use custom events to track that history. We record custom events whenever we: start writing to a file, finish writing to a file, upload a file to object storage, download a file from object storage, modify a file, and eventually when we delete a file that has passed out of retention. The information we track on each of these lifecycle events is highly specialized and specific to the details of operating a distributed database, and immensely valuable for analyzing and understanding its behavior.  As a simple example, if we want to know how much data we stored for each of our customers in the last day, we can write a query like:

SELECT sum(fileSize) FROM ArchiveFile FACET account SINCE 1 day ago

Of course, most of our customers aren't running a distributed database. They all have their own unique industries and markets, whether that's retail sales, online games, banking, or automobile manufacturing. Your custom events probably won't look like anyone else's, but they will fit your needs perfectly.

Custom events are also great for enabling experimentation and data-driven decision-making. Say you have an idea for a new implementation of a subcomponent, but you need to make sure that it actually produces the same answers, and is actually faster in practice. And not just faster on average; you'd better be sure that it's not making 90% of requests better but 10% much worse. What you can do is run them both side by side and record a custom event with the details of each request, whether the answers matched, and how long each implementation took. Now you have a rich data set you can analyze to guide your decision.

You'll also love the performance of custom events. Custom events for specific purposes are typically much less numerous than more generic types of events like spans and transactions. As a result, they’re generally lightning fast to query.

Why not logs?

You might ask, "How are custom events different from 'structured logs'?" At some level, perhaps this is just a matter of semantics or terminology, but to me 'structured logs' sounds like an oxymoron.. Logs at their core are about free-form text and the ability to record anything you want, as long as you can shape it into text. With log partitions and log parsing rules, you do have the ability to extract structure from that free-form text. But if you're starting with structured information, why not cut out the middleman? Instead of emitting a log like

Logger.info("Sold {} {} Widget weighing {} lbs", order.item.quantity, order.item.color, order.item.weight);

and then writing a regular expression to pull that structured data back out, why not just record the structured data directly? By using a custom event, you can record your data in exactly the format you want with the same ease of use as recording a log line.

How to do it?

There are a number of ways to send custom events, but the two most common are through our APM agents or with our Event API.

APM agents

Let's say you're already using the Ruby agent. Recording a custom event is as simple as just adding

::NewRelic::Agent.record_custom_event('WidgetSale', color: 'red', weight: 12.5)

anywhere in your application code. In this example, 'WidgetSale' is the name of the event type, and the other arguments add whatever attributes you want to the event.

The exact syntax differs for each language agent, but they all make it simple to start recording custom events from any application that's already instrumented with New Relic.

Event API

Sometimes you might want to create custom events from outside of an instrumented application. In that situation, the Event API has you covered. The full details are in our Introduction to the Event API, but basically it's as simple as sending an HTTP request with a JSON payload. For example:

[
      {
        "eventType": "Purchase",
        "account": 3,
        "amount": 259.54
      },
      {
        "eventType": "Purchase",
        "account": 5,
        "amount": 12309,
        "product": "Item"
      }
]

Querying

Once you've sent your custom events to New Relic, you can use NRQL to query and add them to dashboards.  For example:

SELECT average(weight) FROM WidgetSale FACET color

Conclusion

Hopefully you now have an idea of why custom events are so popular at New Relic and how you might make use of them to gain a deeper understanding of your own applications. So get out there and unlock the awesome power of this pillar of observability for yourself!