Three Ways To Forward Logs From Amazon ECS To New Relic

Amazon Elastic Container Service (ECS) is a fully managed service you can use to schedule and deploy containerized applications on AWS infrastructure. ECS supports two launch types that determine the type of infrastructure on which your services are hosted: Amazon EC2 and AWS Fargate.

With EC2 you run containerized applications on a cluster of Amazon EC2 instances you manage yourself. Fargate, on the other hand, provides a fully provisioned and managed infrastructure on which you can run containerized applications.

EC2 allows you to access and customize the underlying infrastructure, which (for now) gives you the widest array of options for collecting logs. Although Fargate reduces management overhead, lack of access to the containers' underlying infrastructure rules out a host-based approach to log collection.

In this post, I’ll give an overview of three approaches to forwarding logs from Amazon ECS to New Relic Logs:

1. Install the Infrastructure agent on the container host to forward logs
2. Add a sidecar container to run the AWS FireLens plugin for Logs
3. Configure a New Relic AWS Lambda function to forward logs with the Amazon CloudWatch plugin for Logs.

Note: Log forwarding from the Infrastructure agent and from FireLens is built on New Relic’s Fluent Bit output plugin. You can also forward logs to New Relic from ECS using a standalone instance of Fluent Bit of Fluentd. In fact, don’t miss Pros and Cons of Five Enterprise-Ready Log Forwarding Patterns.

Additionally, each of these methods are dependent on the ECS launch type in use:

Collecting logs from the container host with the Infrastructure agent

By default,ECS containers are configured to use the json-file logging driver to write application logs to /var/lib/docker/containers/*/*.log on your ECS EC2 hosts.Collecting logs from the container host with the Infrastructure agent

The Infrastructure agent can forward these logs to New Relic. Configuration is simple:

1. Install the Infrastructure agent.
2. In /etc/newrelic-infra/logging.d/, create a file called logs.yaml.
3. Add the following to logs.yaml:
   

   logs:
   
   - name: ecs-app-logs # This value is only used to organize your config; it will not appear in log events.
   
   file: /var/lib/docker/containers/*/*.log

After a few seconds the Infrastructure agent will begin forwarding ECS logs to New Relic.

Tip: To run a standalone forwarder, check out the newrelic-fluentd-output plugin.

1. Collect logs via sidecar container and the New Relic AWS FireLens plugin

Log collection from ECS applications running on Fargate is commonly done using a  sidecar pattern. You can add a sidecar container with ECS task definitions to write logs to our AWS FireLens log router. You could also create or update a CloudFormation template to run the FireLens log router alongside application containers.

Protip: You can use log drivers to configure each container in an ECS task definition to log to different destinations. You should configure ECS application containers to use the awsfirelens log driver; however, I recommend configuring the log router itself to write to CloudWatch via the awslogs log driver. This will give you visibility into the log router's container logs, which can make troubleshooting easier if you run into issues during setup.

2.  Fluentd: When FireLens isn't an option

If you’re already forwarding logs to another destination or vendor with CloudWatch or FireLens, you won’t be able to use our custom log forwarding integrations with either source. Containers can only forward logs to a single FireLens log router, and CloudWatch does not allow multiple Lambda functions to be subscribed to a single log group.

In this scenario you'll need to:

1. Configure a shared data volume in the ECS task definition.
2. Configure the applications to write logs to a directory on the shared data volume (in addition to STDOUT and STDERR).
3. Configure a container running a custom Fluentd image. Note that you’ll need to configure Fluentd to tail the logs written by the application containers on the shared data volume.

3. Collect logs via the CloudWatch plugin for Logs

Using the awslogs log driver, ECS containers can write logs directly to a CloudWatch log group. As part of the CloudWatch plugin, we provides an AWS Lambda function that pushes logs from CloudWatch to New Relic Logs.

To enable the CloudWatch plugin for Logs:

1. Install and configure our Lambda function.
2. Create a Lambda trigger to get your logs streaming to New Relic.
3. Optionally configure the number of retries you want to perform if there are network connectivity issues.
4. Generate some traffic and wait a few minutes, then check your account for data.

This is the most "turnkey" method, but it comes with a few drawbacks:

• No parsing capabilities: Data must be parsed upstream at the source or downstream via our platform-based parsing.
• Increased latency: Logs are collected and forwarded further from their source, which adds latency.
• CloudWatch log group subscription limits: Only one Lambda function can subscribe to a CloudWatch log group; so if you’re already using Lambda function to collect CloudWatch logs, you won’t be able to use this method.

Get full visibility into your Amazon Web Services (AWS) environments with a free observability plan from New Relic on AWS Marketplace.