Instrumenting your AWS Lambda functions with Amazon's Distro for Open Telemetry

In this blog post, we explore how to use OpenTelemetry to instrument and monitor serverless applications running on AWS Lambda using the OpenTelemetry collector optimized for AWS, named AWS Distro for OpenTelemetry(ADOT). We look at configuring the ADOT as the Lambda layer for the Lambda function with Node.js runtime and all of the configurations to be deployed using the Serverless Framework.

What is the OpenTelemetry Lambda Collector?

The OpenTelemetry Lambda Collector is a new component that was introduced in the OpenTelemetry 1.0 release. It is a Lambda extension that runs as a separate process from the function and acts as a local proxy for exporting telemetry data from the function to any choice of observability backend.

The OpenTelemetry Lambda Collector has several benefits:

• This simplifies the setup of OpenTelemetry agents for Lambda functions. Users just need to add the collector layer to their function and specify the exporter endpoint as an environment variable. The collector will automatically discover and export the data from the function.
• The collector reduces latency and overhead by batching and compressing telemetry data before sending it to the backend, which reduces network traffic and avoids impact on function’s performance.
• It supports multiple backends and formats for telemetry data, exporting metrics, traces, and logs in various formats (OTel, Jaeger, Zipkin, X-Ray, etc.), all natively supported by the New Relic platform.

What is AWS Distro for OpenTelemetry?

ADOT is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project.

With ADOT, you can use auto-instrumentation to generate traces without modifying your code. ADOT can also collect metadata from your AWS resources and managed services. This enables you to correlate application performance data with underlying infrastructure data, reducing the mean time to problem resolution.

ADOT can be used to instrument your applications running on AWS App Runner, AWS Lambda, Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), Amazon Elastic Kubernetes Service (EKS) on EC2, and AWS Fargate.

ADOT Benefits vs caveats

Benefits

• Simple: ADOT streamlines setup and management with pre-built Lambda layers and integrations.
• Standardized instrumentation: ADOT provides ready-made exporters for popular libraries and frameworks, promoting consistent tracing and metrics across your applications.

Caveats

• Limited control: ADOT offers less flexibility for customizing exporters and instrumentation compared to manual configuration.
• No custom instrumentation: If your function uses libraries or frameworks that aren’t supported by ADOT's pre-built exporters, you may need to manually instrument them.

Pre-requisites

What do you need to follow this guide successfully?

• New Relic ingest license key. (Don’t have an account? Sign up for a free New Relic account).
• Basic knowledge of OpenTelemetry
• Basic Knowledge of AWS

How to use ADOT with the AWS Lambda function?

In this example, we are going to use the Serverless Framework to quickly set up the Lambda function along with an API gateway for the entry point. The lambda function is a simple Express REST API with a few endpoints.

Step 1: Create the Lambda function

You can deploy your Lambda function with the choice of your runtime. In this example, we're going to use Node.js 18.x as our runtime.

org: zmrfzn
app: otel-sls-adot-dev
service: otel-sls-adot
frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs18.x
  region: us-east-1

functions:
  api:
    handler: index.handler
    events:
      - httpApi: '*'

Step 2: Add ADOT Lambda layer

Add the Lambda layer for the ADOT to the function configuration. The layer ARN for the Node.js is arn:aws:lambda:<region>:901920570463:layer:aws-otel-nodejs-<architecture>-ver-1-17-1:1. The region and architecture can be configured as per the supported list here.

We need to wrap the function handler with the OpenTelemetry Lambda wrapper, which is a thin layer that initializes the OpenTelemetry agent and handles the lifecycle of spans. The wrapper is located in /opt/otel-handler, as stated in the official ADOT documentation. You can read more about the configuration here.

In the configurations for our Lambda function to use the ADOT, we're defining a set of environment variables to override the default values. Take a look at the ENV VARS defined in the serverless.yml below for the function.

functions:
  api:
    handler: index.handler
    events:
      - httpApi: '*'
    environment:
        # https://aws-otel.github.io/docs/getting-started/lambda/lambda-js#enable-auto-instrumentation-for-your-lambda-function
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
        OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.nr-data.net:4317
        OTEL_EXPORTER_OTLP_HEADERS: 'api-key=${ssm:/nr_experiments_ingest_key}'
        OTEL_SERVICE_NAME: "Node-Lambda-ADOT"
        # https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/node/opentelemetry-instrumentation-aws-lambda#aws-lambda-instrumentation-options
        OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION: true 
        EXPRESS_OTEL_API_ENDPOINT: 'http://3.230.230.121/v3/api'
    layers:
      - arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-17-1:1

See the full configuration of serverless.yml here.

org: zmrfzn
app: otel-sls-adot-dev
service: otel-sls-adot
frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs18.x
  region: us-east-1

functions:
  api:
    handler: index.handler
    events:
      - httpApi: '*'
    environment:
        # https://aws-otel.github.io/docs/getting-started/lambda/lambda-js#enable-auto-instrumentation-for-your-lambda-function
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
        OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.nr-data.net:4317
        OTEL_EXPORTER_OTLP_HEADERS: 'api-key=${ssm:/nr_experiments_ingest_key}'
        OTEL_SERVICE_NAME: "Node-Lambda-ADOT"
        # https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/node/opentelemetry-instrumentation-aws-lambda#aws-lambda-instrumentation-options
        OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION: true 
        EXPRESS_OTEL_API_ENDPOINT: 'http://3.230.230.121/v3/api'
    layers:
      - arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-17-1:1

plugins:
  - serverless-offline

dashboard:
  disableMonitoring: true

package:
  excludeDevDependencies: true

Step 3: Deploy and explore data

Deploy the function with the OpenTelemetry Lambda Collector layer, it's time to explore the telemetry data from the Lambda in the New Relic platform.

Our Lambda has two different endpoints, which are bridged via API gateway, /path and /weather. Let's take a look at the traces from the second endpoint, which is /weather?location=bangalore

The /weather endpoint in the Lambda function calls an external entity, which is also instrumented with OpenTelemetry. As a result, the distributed traces capture both the entities and the New Relic platform automatically generates a service map that shows the complete journey of the request. Check out the distributed traces and their attached attributes in the short clip below

Conclusion

By adding ADOT as a Lambda layer, you can gain deeper insights into your Lambda functions through rich telemetry data without the need for manual instrumentation. However, you can also choose to manually instrument your functions using the OpenTelemetry SDKs to capture even more detailed metrics. This can help optimize your Lambda functions and other serverless applications. If you're interested in manually instrumenting your Lambda functions, check out this guide to learn more about using the OpenTelemetry SDKs.