Elasticsearch is a search and analytics engine that has become one of the most popular distributed document stores since its release in 2010. It's an open source tool that delivers lightning-fast and highly relevant search results to users. If you’ve searched for a show on a streaming service or ordered groceries from an app for delivery, chances are Elasticsearch was part of that journey.
Monitoring Elasticsearch in context with the rest of your Ruby application can help you identify and solve performance issues and errors related to requests to your Elasticsearch clusters.
Version 8.12.0 of the New Relic Ruby agent introduces automatic instrumentation of versions 7.x and 8.x of the Elasticsearch gem, giving you a more complete picture of how your Elasticsearch implementation is working over time. Because we’ve automatically instrumented Elasticsearch, you just need to upgrade to Ruby agent version 8.12.0 or later to monitor requests from your Elasticsearch client.
In this blog post, you’ll learn how our team instrumented Elasticsearch with Ruby metaprogramming, how to update the Ruby agent, and how you can configure the Ruby agent to capture or obfuscate queries as needed.
Instrumenting Elasticsearch with metaprogramming
The New Relic Ruby agent uses metaprogramming techniques to insert New Relic instrumentation code into commonly used public methods of the libraries we want to monitor. Many libraries, including the Elasticsearch gem, aren't designed to use callbacks or hooks. To work around that, we use metaprogramming to add custom instrumentation methods to these libraries.
We automatically default to using a technique called Module#prepend to insert instrumentation code in the ancestor chain of an existing method but lean on alias_method chaining when a conflict occurs between these two techniques. Here's an example from the source code that uses chaining.
to_instrument = if ::Gem::Version.create(::Elasticsearch::VERSION) <
::Gem::Version.create("8.0.0")
::Elasticsearch::Transport::Client
else
::Elastic::Transport::Client
end
to_instrument.class_eval do
include NewRelic::Agent::Instrumentation::Elasticsearch
alias_method(:perform_request_without_tracing, :perform_request)
alias_method(:perform_request, :perform_request_with_tracing)
def perform_request(*args)
perform_request_with_tracing(*args) do
perform_request_without_tracing(*args)
end
endThe Ruby agent offers both instrumentation options to give you greater flexibility. New Relic uses Module#prepend by default, but if you have another gem included in your application that uses method chaining on the same methods as the New Relic agent, errors can occur. That’s because method chaining (alias_method) and Module#prepend work together only in particular situations. When not used together properly, they can cause non-terminating recursions. Our troubleshooting guide for the SystemStackError provides details on these two options if you have a conflict with one of the instrumentation methods.
The next image shows the default option for auto-instrumentation of the Elasticsearch library in New Relic. See the Ruby agent documentation for more information.
The New Relic Ruby agent inserts the instrumentation code into the perform_request method at runtime because the Elasticsearch ruby gem routes all API calls through this method.
The New Relic Ruby agent supports versions 7.x and 8.x for the Elasticsearch gem.
The perform_request method doesn’t include information about the specific operation type, such as update, delete, and so on. To solve this problem, we found that we could check the caller_locations in order to see which method was responsible for invoking perform_request.
Here's the code where caller_locations is used:
def nr_operation
operation_index = caller_locations.index do |line|
string = line.to_s
string.include?('lib/elasticsearch/api') && !string.include?(OPERATION)
end
return nil unless operation_index
caller_locations[operation_index].to_s.split('`')[-1].gsub(/\W/, "")
endThis provides us with the specific operation being called on Elasticsearch, so we can attach the correct operation type to spans.
Here's a screenshot of a distributed trace of Elasticsearch spans in New Relic.
Elasticsearch spans have an attribute, db.instance, that displays the name of the cluster the Elasticsearch request is sent from. This can help identify if there are any specific clusters encountering errors or performance issues. Spans will also record the query parameters passed to perform_request, which are obfuscated by default to protect sensitive information.
And this screenshot shows the attributes recorded for an Elasticsearch update span, including attributes db.instance for the cluster name and db.statement for the query parameters.
Elasticsearch configuration options
We included two configuration options, both of which are enabled by default: capture_queries and obfuscate_queries.
When elasticsearch.capture_queries is enabled, the agent will attempt to record the query being passed to the request. We also added obfuscate_queries to keep sensitive information hidden. These options can be configured through your newrelic.yml file or environment variables.
This screenshot shows the configuration for elasticsearch.capture_queries in New Relic.
The next screenshot shows a New Relic Ruby agent configuration for elasticsearch.obfuscate_queries.
Upgrade your Ruby agent
You can start monitoring Elasticsearch by upgrading your Ruby agent. Version 8.x of the New Relic Ruby agent includes new instrumentation for threads Tilt (8.2.0), threads (8.7.0), gRPC (8.10.0), and Ruby’s Logger class with logs in context (8.6.0), along with some bug fixes and enhancements like code-level metrics (8.10.0).
Connect with us
The Ruby agent is open source software. We welcome contributions to our source code and improvement suggestions, and we always look forward to connecting with the community. We’d love to hear about what you like and what you want to see in the future. Connect with us on GitHub.
Next steps
If you’re not using New Relic yet, get started with New Relic for free. Your free account includes 100 GB/month of free data ingest, one free full-access user, and unlimited free basic users.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.
