Instrumenting serverless in AWS

Serverless applications are critical for cloud-native customers, and gaining visibility into their functions can take time and effort. This blog post will offer tips and code examples for engineers using AWS to set up serverless instrumentation easier with New Relic. The example three-tiered application includes API Gateway, Lambda, and Amazon Relational Database Service (RDS). You'll learn tips for instrumenting your serverless functions and debugging strategies for setting up serverless instrumentation using Lambda layers or manual instrumentation in New Relic. 

Whether you are new to New Relic serverless monitoring or looking to optimize your existing setup, this post can help you achieve your observability goals with New Relic. By the end of this post, you'll have a deeper understanding of serverless computing. You'll be equipped with the knowledge and tools to set up serverless instrumentation for AWS quickly.

Why is monitoring your serverless applications important?

Monitoring serverless applications is crucial for identifying and responding to issues, optimizing resource utilization, and improving performance. With serverless architectures, you can’t directly access the underlying infrastructure, so monitoring is essential for detecting problems and taking corrective actions. You can optimize function architecture by monitoring resource usage to minimize usage and adjust allocations to match workload demands. Performance metrics also provide insights into bottlenecks and areas for improvement, such as optimizing code or increasing allocated resources.

Setting up serverless monitoring

This step-by-step guide to setting up New Relic serverless monitoring in AWS using Lambda layers and manual instrumentation uses NodeJS as the language, with Postgres running on AWS RDS.

Prerequisites:

• Make sure you have the AWS CLI.
• Set up the Serverless Framework With AWS.
• Sign up or log in to New Relic.
• Node.js V16.x.x
• RDS Postgres database or a database of your choice. This Git repo has an SQL dump file to import into your database. This blog post does not delve into RDS creation and database imports .
• Git (optional)

Requirements:

• AWS credentials
• A New Relic ingest license key 
• NodeJS v16.x.x

Metrics Stream integration setup

To start, we recommend installing the New Relic AWS integration to gain valuable insights into your AWS services. To monitor all CloudWatch metrics, install the Amazon CloudWatch Metric Stream integration, which monitors all metrics from all AWS services, including custom namespaces. You can use additional integrations to gain extended visibility into key services beyond the available CloudWatch metrics.

Step 1: Set up the AWS integration: Head to New Relic, and select Infrastructure > AWS > Set up AWS Integrations through the UI.

Step 2: Select Use metric streams.

Step 3: Within AWS Identity and Access Management (IAM) , create a role and establish trust with New Relic.

Step 4: Now, attach a ReadOnlyAccess Policy to the role.

Step 5: Set the role name to NewRelicInfrastructure-Integrations.

Step 6 (optional): Configure the Budget Policy to allow New Relic to capture service consumptions and usage costs for the budgets you configured in AWS.

Step 7: Add your account details to New Relic.

Step 8: Within Metric Stream in AWS, configure the Metric Stream and get your New Relic ingest license key under API Keys.

Note: Enrich your metrics by setting the resource metadata from AWS Config to true.

Step 9: After the stack starts creating, go back to New Relic and select done for your integration. Select the account status dashboard to view your Metric Stream Status.

Step 10: In New Relic, select All Entities in the menu. You'll start seeing your available AWS services as entities.

Note: It will take a few minutes for all of your AWS data to show.

Step 11: After it's installed, you may notice Lambda entities being created if you have existing Lambdas, because New Relic has picked up your serverless functions. Keep in mind you are capturing Lambda information, but you have not yet instrumented the lambdas. The next section uses a simple lambda function generated by the serverless framework (SLS) querying a database in AWS, where you'll see database calls with New Relic tracing.

Lambda instrumentation

Step 1: Clone the repo, or use the lambda of your choice.

Run the next command to clone the repository. If you want to use another method, go to the Github repository.

git@github.com:Tirslo/serverless-nodejs-101.git

Step 2: Encrypt your credentials.

Before instrumenting your Lambda, ensure you have uploaded your API keys/credentials to the AWS Parameter Store using AWS Systems Manager. If you're testing a dummy function, you may skip this step.  But we strongly advise you to encrypt your credentials.

You can easily encrypt your credentials from the CLI using AWS Systems Manager by running this command:

aws ssm put-parameter --name "pgPortDevRel" --type "String" --value 5432

Repeat these steps for your database credentials and New Relic credentials. You can find your New Relic ingest license key in API keys. 

Step 3: Set up the  serverless instrumentation.

• To begin instrumentation, open your preferred code editor and navigate to the repository. We suggest using Visual Studio Code. Among the files available, you'll only be working with the serverless.yml file. Open it to proceed.

Setting up this configuration is simple.

• Begin by creating a service and specifying your cloud provider, lambda runtime, and region.
• In the functions section, create your "blog-function" and pass the necessary environment variables for your database credentials, host, and port.

Note: This references the AWS credentials from Systems Manager.

• This example specifies two events that will occur when someone accesses the function through API Gateway. A GET request will show a list of games stored in the database, and the POST request will add a game.

Note: API Gateway is deployed when you've specified your events.

So how do you instrument your code from here? Do you have to add our library to your code, or can you auto-magically do it with lambda layers? If you're using Node, Python, Java8.al2, and Java11, there's no code change required to instrument your lambdas! For other languages and runtimes, you’ll need to follow this manual instrumentation.

For the specified languages, the lambda layers :

• Do not require any code changes in New Relic.
• Enable New Relic APM agent functionality using a single layer.
• Configure the New Relic Lambda Extension and New Relic License Key managed secret automatically.
• If the extension is disabled, this plugin configures CloudWatch log subscriptions and subscription filters automatically.

This example uses Node 16.x. Add your serverless plugins. To instrument the lambdas, all you need to do is install the newrelic-serverless plugin.

With NPM:

npm install --save-dev serverless-newrelic-lambda-layers

With YARN:

yarn add --dev serverless-newrelic-lambda-layers

• After it's installed, modify your serverless.yml file with this code:

• Deploy your Lambda using sls deploy.

It’s that easy to instrument your lambda with the serverless framework.

Step 4: View your instrumented Lambda.

• Generate some data.
• Go to New Relic and select all entities. In the Lambda functions section, you should see your Lambda function with the instrumented column YES.

• By traversing Lambda tracing, you can identify the queries made to the backend database. The entity map identifies your dependencies. If they aren't instrumented with New Relic, you'll still see a dependency link to an “external service”. Take note of the query analysis in the right pane. You can see the query run and the duration, further broken down in the spans below, showing each step and duration of the request.

• At the top left, you can see a Logs tab. New Relic automatically correlates your logs in context of the inspected trace. This allows you to identify and traverse potential issues sooner, as you don’t need to bounce between different UIs to access your relevant logs. Logs in context provides contextual logs— not just on the selected service but all linked dependencies if available.

Once your lambda is instrumented nicely, here is the end result with New Relic.

Top troubleshooting tips

If you haven't set up your instrumentation correctly, the instrumented column will be blank, or your Lambda will not be viewable in New Relic.

"Function not in New Relic"

• Ensure you have set up the Metric Streams integration correctly. Check your account status dashboard New Relic > Infrastructure > AWS > Account Status Dashboard for errors.
• Ensure you have used the New Relic Ingest License Key → API Keys.
• Check your Lambda Execution Logs for any errors. You should see something similar to this output:

"Lambda runtime not supported"

• Check our Lambda Layers to see if your runtime is supported.
• If your runtime is supported, check your logs in New Relic for further details. You can access your lambda logs directly from your function in New Relic.

Explore your instrumented Lambda

Properly instrumenting your Lambda function is an essential step towards gaining better visibility into your serverless functions. With the right instrumentation, you can access a wide range of views that provide critical insights into your serverless functions. You can observe the performance and behavior of your Lambda function and detect anomalies.

If you opted in for the Lambda layers, notice that without any code-level instrumentation, you can access logging, tracing, and Lambda metrics. These tools are critical for monitoring your Lambda function and detecting issues before they escalate.

By leveraging these tools, you can quickly identify performance issues, locate the source of errors, and optimize your function's performance. This level of visibility is vital to ensuring the reliability and scalability of your serverless applications.