These are the top 10 snapshot results. To see complete query results, hit 'Return'.

Webinar:

Scaling with Docker: New Relic's Containerisation Journey

 

We'll admit it. Our success was killing us. New Relic has great Saas-based APM software, but we were struggling to provide fast response time across trillions of event queries a day. Our move from monolith to a modern, scalable, software stack provides many lessons for enterprises making similar journeys.

Watch the recording!

In this webinar, we'll discuss:

  • New Relic’s migration to containers

  • Is Docker ready for production?

  • How do you know what’s happening with your containers?

  • Maintaining your success



Full Script

Moderator: Hello, everyone. Thank you for joining today's webinar, Scaling with Docker: New Relic's Containerisation Journey. You'll hear from Tori Wieldt, developer/advocate at New Relic.

Before we get started, just want to go over a few housekeeping items with you guys. Today's webinar will be recorded. A recording will be sent out to you shortly after the presentation.

There will be time reserved at the end for question and answers, so please be sure to place any questions you might have in the Question panel of the GoToWebinar control panel. Now, I'll hand if off to Tori.

Tori Wieldt: Hello, everyone. Good morning, afternoon, wherever you are. I'm delighted you joined us. I wanted to share a little bit about what we've learned at New Relic.

I do have to start with the obligatory Safe Harbor slide. I did put a typo in there. I'm challenging you to read this entire slide and find the typo. All right, just kidding. What it pretty much means is, don't make decisions based on what I've told you, although I'm going to give you some really great information, so maybe you should.

I wanted to share with you our journey to containerization. What we learned along the way, how we went from a monolith to a modern, scalable software stack, and hopefully give you some lessons and some insights into the things we've learned, and help you along the way on your journey.

I'll share what we've liked about Docker, what's worked for us really well. I will answer the controversial question, "Is Docker ready for production?" I'll tell you some of the challenges and limitations we've encountered along the way. How to look at what's happening inside your container, this is key.

You've got to know what's going on with your containers. How do you maintain your success? You've moved to a services architecture, how are you going to have the most success using Docker in containers? For those of you that don't know us, New Relic is a cloud-based software analytics company. We give visibility to our customers, into how their applications and their business is running.

We do this through a variety of agents, which are installed on the applications, and people use it to monitor their services in the cloud. All this comes together in the New Relic cluster, where we collect all that data, we process it, we store it, and we make it available for people to query in a fast, ad hoc manner using our Web and mobile interfaces.

The bottom line here, two bottom lines is, one, our application in the cloud is our product, so if we're not up, we have nothing to offer our customer. Also, we're dealing with lots and lots of queries a day, so that is a T for trillion. Roughly, and this may be updated, but we deal with over 100 trillion queries a day.

What was our journey like, what can we learn, and what can we share with you? We started out with the classic Monolith. It was a Ruby app, it was a single application, it ran our entire business. It contained the agent, the data collection timeline, and the Web interface. Back in ancient times, when it was divided out into the duo list.

This is with a Ruby on Rails application, which was the user interface, and then the back end was a Java collection pipeline. As our customer base grew, and our feature set grew, and our company grew, we started to have problems. Problems which are probably familiar to many of you.

As our success continued to grow, our problems continued to grow. How did we think about solving that? We moved to a services architecture. We looked in and decided we're going to have people create services, and we thought, "Oh, we might have dozens, or maybe at the top, 100."

The reality is, we're running 200 plus services, and we needed a way to deploy it faster, so we chose Docker. Keep in mind, we chose Docker in January of 2014. Those were the early days, and perhaps that was a little ambitious, but Docker had other characteristics we wanted.

It isolated the experience of developing applications from the maintenance of that service. It provided great efficiency, and good uptime. We looked at our office people and said, "Hey, instead of running two or three services, you have to run 200."

Docker provided a more standardized interface to let them do that. Here's how deployments worked before and after Docker. This is the generic view, it's not 100 percent specific to New Relic, but it'll give you a sense of the change that we went through. It's hard and expensive to get communications and processes right between teams of people, even in small organizations.

Detailed communication has to go between the development team and the operations team. You can see on the left, the development team would request a resource. The ops team would provision the resource. That would often take a long time. Then the development team would script the resource, and you would tweak the deployment. Of course, you had loop through this several times in terms of fixing problems.

You are probably all familiar with dependency hell, knowing what rev, what patch of your database, of your OS, of all the libraries that your app is using can be really challenging. With Docker, we're able to reduce that to a lot fewer steps. I don't want to create the illusion that there is no need for an operations team, and they don't have anything to do.

The graph on the right is pretty simplified. They still have to work on provisioning, and working on optimizing the infrastructure, and that's a key piece that we need to have done, but a tool that reduces the complexity of the communication, while aiding in the production of more robust software was a big win for us. That's exactly what we've found with Docker.

Now, Docker's no panacea, and implementing Docker takes work, but it's a good approach to solving some of the real-world organizational problems we had, and it helps us ship software faster. This is key. Designing a well-designed Docker workflow can lead to happier technical teams, and real money for our bottom line, which has been definitely our experience.

How do we use Docker? We use Docker for almost everything that New Relic serves up. There's some specific high throughput services that we don't provide on Docker, but for the most part, our whole stack is containerized. Here's a good example.

We have an application called New Relics Synthetics. We like to call it a crash test dummy for your software. What Synthetics does is, based on Selenium, and what it does is it can ping your application from all over the world. It can also work on simulating a login to your application, or somebody checking out something from the car.

Anything that's key for your application to use, you can synthesize that, and have a script do it for you on a regular basis. This is a key way we use Docker, because each script that a user spins up runs in its own container. We use this both for performance, and security reasons. Performance-wise, containers spin up so quickly. That's important for us.

Security-wise, it provides a much smaller, what we call it is an attack vector. That container is up, does its job, and goes away. It's a lot harder for somebody to hack that. We've been using Docker for this, and what we've seen is a lot of people using Docker in the specific way. The container spins up, does a small job, and then goes away.

What we'd like to call that is the ephemeral infrastructure. In this slide, I'm generally talking about, New Relic is in an interesting position. We use Docker ourselves, and also our customers use Docker, so we're able to look at how our customers are using Docker. We do see that customers, yes, they use Docker as a VM.

Just replacing a VM, doing a listing shift, but also they start using Docker in the cloud, in the EC2 environment. There's also this short-lived containers. A container is a process, if you will, lambda-ish, where the container comes up, does its job, and goes away.

We've found that fully 25 percent of our customer's usage is containers that last for less than five minutes. This is interesting in terms of making decisions about your architecture and infrastructure for containers that only last a short amount of time. What have we found in Docker that we like? Why does New Relic love containers?

Our developers are productive much faster. There's less yak shading, more time spent coding. I have to be honest here. This did not happen overnight, and I think the developers would personally come and throttle me if I didn't explain that it was bumpy in the beginning, but now that we have better tools, better way to support the developers, it does get them up and running much more quickly.

We can get people provision in the matter of hours, rather than weeks, and that's really important. Also, as I showed the streamlined communications between teams, it means less handoff, less errors. This is very important in terms of the title information that has to go back and forth between teams.

If you can get that organized in a better way, that's very helpful, and we like containers because they're better than VMs. Containers launch in nanoseconds. They're much faster than VMs in terms of launching, and utilization is four to six times better than VMs.

Since a separate kernel doesn't load for each user session like it does on a VM, the overhead associated with multiple OSes don't happen in containers. This is why containers use much less memory, and CPU, than VMs running similar workloads. For example, you can't put 100 VMs on a Mac, but I have run 100 containers on it.

Docker allows for scale. The abstraction allows for scale. We're talking about scale in several different ways. The key way people think about scale is input. How many users do you have? What's the throughput of your app? How many times are you getting ping? In reality, there's different ways that you have scale, and you have to think about it.

One is the size of the app itself. Another way is the number of features on your app. What are you providing to your users? You have to think about scaling up that way, and scale in terms of number of engineers that are working on your application. How can you scale that up?

One is, you want to get them productive quickly, but in another way, you want to make sure that what they're working on when they introduced changes to their application, their service, they're not causing disruption throughout the entire system. Abstraction allows for scale.

Docker allows you to easily deploy new containers, programmatically. Also, our system is elastic. It can grow and shrink with the load, automatically. Different pieces of your system, as I mentioned, can be swapped out, scaled up without affecting the rest. We can put out releases without downtime.

Why do the ops people love containers? There's no more activating, kickstarting of pizza boxes. This is not only a Docker thing, this is thanks to the cloud, but ops people don't have to get in there and physically spin up a machine. You just script what you need, and build once, run anywhere works for infrastructure.

As one ops person told me, ops is now Costco instead of the corner store. They're providing a big chunk of ice instead of special snowflakes. Also, we do an exercise with disaster recovery. As I said, our application is available. It's in the cloud, and so if we don't have an application up, we don't have anything to offer our customers.

We've done disaster recovery exercises, where we build all the services from scratch. 80 percent of the containers took 20 percent of the time. Those were the standardized Docker image, I'm going to put it out there, and then the special snowflake containers, those 20 percent of the special snowflakes took 80 percent of the time.

That's classic, but Docker did allow for better scripting, and better rebuilt, and for the disaster that is not going to happen, but it's good to be ready for it. Docker provides much less fiddling with Puppet and Chef. Now, developers own those dependencies, so they are responsible for figuring out what their dependencies are for their particular service.

Another thing that's great about containers, both development and ops, you don't spend a lot of time troubleshooting your container. What you want to do is just replace it. I have a friend that says, "Treat them like diapers. If you're having a problem, throw them away. They're cheap to spin up another one."

Why do developers love containers? Well, the faster provisioning is an important piece, not only businesswise, but if you think about the human part of it, I worked on this new functionality, I want to get it out to users as soon as possible. And so, telling the developer, "Oh well, it's going to take two weeks," it can be done in minutes.

Developer enthusiasm is pretty cool. When they worked on the future, they can get it out, and they can start getting feedback on it almost immediately. Developers also control their runtime environment. You own exactly what the app gets. Tool redundancy is "easy." I put easy in quotes because it's not 100 percent out of the box, doesn't require any tools.

There does have to be op support around this, but it's easy to put your image out, put a second image right next to it, run exactly the same thing, and then best practice is strap it across hosts. You want to have four images of your service running, two per host. That way, that redundancy helps with availability, in case a container goes down.

Docker is very microservices friendly, and it helps with the way of looking at your container. You can start thinking about architecting one service per container, and that can be a best practice that you can use. It's a great way to start thinking about, "Oh, I have a login service? That goes in one container. I have another feature? That goes in another container."

Another thing we've seen with Docker is, because these containers can run independently of each other, then you can pick the best language for a job. Not everything has to be in Java, not everything has to be in Rails. You can use the best piece. We have pieces of our application that are written in Go. For that particular functionality, Go is the best language, and that's what we use to support it.

The controversial question, is Docker ready for production? People have been asking us, "How long has Docker been out? A couple years now?" Everybody wants to know. Well, it's a trick question, because we are using it in production. Ready or not, we've gotten a lot of functionality out of it.

We've gotten a lot of good things that are happening, but it is challenge. It is a dynamically moving technology that we're using. Docker is a moving target, and pretty much, if you think about Docker as a system API that's strapped to things in the Linux kernel, so Docker doesn't have complete control over what they're using. If there are issues with the Linux kernel, that has to be corrected.

I do have to say, Docker is doing a great job. With each release, it's getting better and better. We're seeing better tools, better security, things like that. It is improving. I would say that I would stay away from being on the cutting-edge. [laughs] Your mileage may vary, but I'll talk about that in a minute. Also, wow, "Is Docker native on Mac and Windows?"

According to the latest release, it is. There's been the transition from using boot to Docker, and now Docker Toolbox, and now it's native. That's going to put some bumps in the road. If you're developers, most of our developers are in Mac, so it's going to change the way they work with Docker. Be ready for change.

Also, one of the challenges with Docker is the docks are written for green field development. There's a great Docker "Hello World" tutorial out at docker.com. I'd recommend taking a look at it if you're totally new to Docker, but it doesn't provide information on how to take an existing app and Dockerize it.

Most of that image, now, is in trivial form. How to Dockerize your data center. I haven't seen anything yet. I'm sure there's probably some blogs out there, but there's nothing that's detailed that tells you how it's going to do that. As the ops people have told me, Docker is lots of pain, but we do get a lot of gain.

One of the biggest challenges is getting that workflow into everybody's lives. What are the challenges and limitations to Docker? As we said, it's not a panacea. It's not going to cure everything. Here's some of the specific challenges we encounter. There is a learning curve of, people are going to have to learn the new tools, go through dev and ops. We'll need to acquire new skill sets.

On the outside, you'll need a solid library of base images, a process for building and shipping Docker, and you do need good internal docs and support for developers. This is part of the bumpy road when we roll out Docker, and didn't provide tools or information. It was frustrating for the developers to use it.

On the developer side, you have to learn those tools that have been provided to you and Docker, and you have to take ownership of your app's dependencies, and you have to let go of certain things, like having access to a persistent file system. Logging your app, you can't just log your app inside the container. It's going to fill up the container.

You have to start thinking about your app in a little bit different way. Also, as I mentioned, legacy apps. We found that converting legacy apps to Docker is a pain point. It's great once you know all the dependencies on your app, and you could build containers repeatedly, but sussing out those dependencies can be a tough exercise.

Once your application is running in a container, there are things that you take for granted, that you have in your own environment, that don't exist in a container. For example, that you can write to a temp directory, that just doesn't exist in your container.

Also, multi-threaded code. You can use the Docker scale command to easily create additional images of your additional containers of your images, but that doesn't make your code multi-threaded. You have to code your app to be truly redundant. Also, there is a noisy neighbor problem.

Containers are isolated from each other, but it's probably more limited than you might expect. While you can put limits on resources, the default container config has been sharing CPU and memory on the host. This means that unless you constrain them, containers will compete for resources on your production machine.

This is one of the gotchas that we encountered, is we did not put memory limits on our containers, so as the best practice, you absolutely just need to put some sort of memory limit, so it doesn't run away and cause problems for all the containers on that same host. There are issues with security on Docker.

Our ops people use base images to enforce some best practices around security, like preventing anyone from logging into a container is used. Patch management is also an issue. When we discover a vulnerability, we have to upgrade our whole environment quickly. It's obvious you have to patch the Docker host, right?

You have to update your Docker files for future revs of that host, and then you have to remember that developers are also running that whole environment on their laptops. You have to get them to pull the updated base images onto their machines, and it's a new habit that you have to build into your organization.

The limitations of Docker. Docker is great, it does a lot of things, but it is a dessert topping, and a floor wax. It doesn't take care of everything for you. What's your host? That's a decision you have to make for yourself. Docker is not the cloud. You can run Docker almost anywhere, and that's your decision about where you're going to run it.

You have to choose your cloud platform, if you're going to be in the cloud, and figure out what works best for you. Docker can solve a lot of things, but sometimes you need a specific tool with more in-depth features than Docker can offer. Docker can significantly improve your organization's ability to manage applications and your dependencies, but it does not directly replace traditional configuration management.

We still use Puppet. We also use Vagrant to support testing on images that match our production environment. Docker by itself can't automate your deployment process. We've deployed hundreds of times a day, and we started using Docker early on, and there were hardly any tools.

We built a tool that's called Centurion. It's a Docker deployment school, and with that tool, the build shifts containers to the Docker Registry, and then Centurion sends containers out to the Docker fleet. What it does is, it uses your Docker server form as a deployment target, so it handles things like volume mounts and port mapping.

We have open sourced Centurion. We're sharing it with the community. It's out on Github, under New Relics Centurion. Please take a look at it. If it helps you, you're welcome to use it. Being an open source project, we'd love for you to contribute back, if there's some code or some feature you'd like to improve on it.

How do you know what's happening with your containers? You have this great new delivery system, where you're delivering quite quickly. You have this ephemeral infrastructure. How do you know what the heck's going on? At the most basic level, there's something called the Docker Steps command.

This is a command line tool, and it gives you the basic resource consumption of your containers, and it'll tell you the names of the running containers. It'll present the CPU utilization, the memory, the total memory. Note that if you do not have limited memory for your containers, this will post the total available memory of your host, and that's not the total available memory to your container.

That doesn't mean that it has that access, but you, listening to me, make sure that you're already constraining the memory of your container, so this won't be an issue for you. In addition, you can see the total data sent and received over the network by the container. The step command is OK. You can also use the Docker API.

It's very similar to Docker Steps, and it can give you a live stream of what's going on with the CPU memory, and it provides a little more detailed steps than the steps command itself. How can you attach to a running container and see what's going on inside of it? These two commands are used by our ops people.

The first command, obviously you have to plug in your Docker host, and your port number, and the name of your app, but this graph command will give you, returned back a container ID that you can then use in the next command. That will connect you to your container, then you're running a bash shell inside the container.

That way, you can just troubleshoot in a way that you usually do, but we're running Docker at scale, so we don't want to look at individual containers. We don't want to spend time looking, tweaking around with individual ones unless we absolutely have to.

This is where I want to show you what New Relic does with Docker containers, and what information it can provide you. To provide some sort of contacts, our basic product is called APM, application performance management. This is a screenshot of the overview screen.

It gives you basic information about what's going on with your app. We want to look at your app, and see what's going on. If you scroll down on this particular screen, the overview screen, you're going to get a sense. You're going to get a table like this, that's going to tell you what's going on with your containers.

It's broken out by server, and lists each app instance in the cute little, I don't know, does that look like a Lego? The Lego icon there provides for you. It shows you that it's an individual container, and it's group by server. I've looked at an overview of my app, I see a server that I want to drill down on, and I want to get some more information on.

Each one of those is a clickable link, so if we go and look at our Docker form, it takes us over to the New Relic server product. This gives you basic information, similar to Docker Step. It gives you CPU usage, what's going on with physical memory, load average. This is the overview for a particular server.

This is interesting, but I don't think it's valuable. We believe you should focus on what's going on with your applications, instead of a particular server. This is what we provide in the server Dockers tab. It's broken out, it's rolled up by images. This shows us by CPU, the largest consumers of CPUs.

It's just a stack rank. This is what you care about. You can see the top list here. This image is running in two different containers. We combined it, and the CPU consumption is seven percent, nearly eight percent. The memory is around 30 gigabytes.

This is the detailed information, apps or services are roughly equivalent to images, so this is a great roll-up. This is a quick way to give an overview. We've used this a lot. It really minimizes the finger-pointing. If you're introducing a new service out on a container, and people are like, "Oh, you've just destroyed my system, or my host," this is a really good way to take a look at it and go, "Is what I introduced a problem? Perhaps it is not."

I hope that was helpful, giving you an idea of what our journey was, how to look at what's going on with containers. How do you maintain your success? What are some of the key takeaways we provide for you? One, I'd say make sure you start simple. Don't start with your most intense, high throughput app.

A great way to start is if you're adding new functionality to your application. It's a great way to break apart your monolith, use it as a micro service, and you can use that particular service in a container. Also, we found that a lot of the benefits we got from Docker came from organizing around the technology.

As I described, our whole workflow has changed. The way we interact, the hand-off between dev and ops has improved, and so we've gotten a lot of benefits out of looking at one technology, everybody across the company using it, and centralizing on that.

Of course, it's key to keep a good watch. Monitor what's going on. It's very important to look at what's going on with your application. You can do infrastructure in, or infrastructure up, but we think it's really important to know what the application is doing, therefore what is your user experience?

Nothing lasts forever. You have to architect for the ephemeral. Remember, I said that 25 percent of our users, their Docker containers last less than five minutes. What does that look like for you? How do you offer services where you can spin off a Docker container, and it goes away?

Also, nothing lasts forever. Docker's great, Docker's going to change a lot. Docker's going to get modified, and it may be replaced by another technology. That's the joy we get of working in a technology field. Things are always changing, so don't get too attached to it. It's very helpful, we've gotten a lot of benefits out of it, but keep in mind that there will be a new technology coming down the road.

Thank you for joining me. Here's some resources in terms of learning more about Docker. Docker does provide a nice "Hello World" tutorial that you can go through. We have two engineers who wrote the book, "Docker and Production," and that's been very helpful for people to take a look at.

I really appreciate the book because it's a nice look at the theoretical part of Docker, but it does offer some cookbook instructions on how to work on Docker, and scale it up. If you go to newrelic.com/docker, you can get the first three chapters of the ebook for free, so please take advantage of that.

There's a nice Docker tutorial, Docker for Java, for Java developers, developed by Arun Gupta, who's over at Couchbase. He was able to put this tutorial together, and share it with the community. It's out on GitHub, take a look. It walks you through the basics of Docker, how to work through multiple hosts, and also the tooling that's available on Eclipse where you can work with Docker directly through your IDE.

Thanks. I'm really glad you attended. I will be posting these slides to slide share, and you will get a copy of, you will get an email that tells you details. If you have any questions now, we'd love to hear from you, or also you can reach out to me on twitter. My twitter handle is my name, @toriwieldt.

Thank you. What questions do people have? Let's take a look. The first question is where do you put the configured information for the deployment information? That's outside of the container, in your Docker file. That's where you put it.

Natasha Litt: Hey everybody, I'm also a colleague of Tori's, Natasha Litt. I have some flight hours, both on the operations side, and on the development side with Docker. I'm joining in to help answer any technical questions. Tori, mind if I chime in on this one?

Tori: Absolutely.

Natasha: Great. The configuration information for each deployment environment goes in, definitely the Docker file has some configuration specifics for your app, but the way we try to approach it is basically that your runtime app equals your built container, plus a bunch of environment variables that you feed into it.

For example, like in staging, maybe you're talking to a staging database, and then in correction, you're talking to the big iron database. What we'll do is we'll boot up the Docker containers on our staging environment by feeding them environment variables that point the app to the staging database.

Whereas, the same container is fed a different set of environment variables in production, and that makes you talk to production instead. The open source tool that we wrote, Centurion, makes it really easy to basically create a little portfolio of environment variables that get fed to your container, in different environments.

That's in our own ecosystem, we use Centurion configuration files, rig files to store the configuration information. That makes it really easy to organize very large sets of, where we've got hundreds of repos, and we can still know where everybody's staging and production environment variables go.

That ties back into what Tori was saying about, the work it takes to Dockerize your app. developers have to take the time to find every hard-coded reference in their code, and extract it out into the environment so that it can be controlled from the environment, so that the image can run in production, or in staging either way, just depending on changing the environment variables.

Tori: Wow, great. It's great to have the pros who have been in the trenches to help out with that, thank you. Let's see, Chris has a question. I presume there some agent to be installed for APM. Is there a Docker container for that? What you do with the agent as you install it on your application, on your running app, that's how you get the APM information.

Then you have to do some sort of tweaking on the server side, and that's in our documentation. That way, servers talk to APMs, and that's how you can get the clean roll up information on Docker. Oh, Natasha, I'm glad you're here. Next question. If you have a private Docker registry, what does your naming tagging strategy look like for images?

Natasha: What a great question. We have a sort of private public Docker registry. Because the Docker registries out there, Docker Hub and key.io and that sort of thing, we like them, but for security reasons, we wanted to have very fine-grained controls over who could access what.

We created our own, we basically wrote some open-source software, it's called Dogestry. It turns an Amazon S3 bucket into a registry. We use that as our private Docker registry, except that the software we used to privatize it is public, so you could use it yourself, if you'd like.

What does the main tagging strategy look like for Image? Great question. What we do is, every time you build a Docker image that is a release candidate, you tag it, and you push that tag to the repo, or to the Docker registry. Then, whatever is the top of the tree you tag latest.

That way, you always know what's running on, what is the next thing to deploy. It's the latest one, and then if you need to revert to a certain shot, what we do is we tag each release candidate with the SHA of the last commit. You can easily look at your GitHub repo and say, "Oh, I need to revert to where the last commit was A3095," and so then you can just push the Docker image A3095, and make that the latest one. Now, you've rolled back to that state.

Tori: Excellent, thank you. We've Got another great question here. Robert asks, did we do testing to measure the overhead delay caused by monitoring apps and containers? I can take that. When we measure to check this out, overhead runs between two and three percent.

Next question. Since billing is per host, does New Relic know if multiple Docker containers were used on a single host? You need a New Relic agent on the host, and scripts agents within the containers. As we discussed before, the agent sits in your app, so if you're running a Java app, you just download the new Relic agent, deploy it, and then it really doesn't matter where it's running, it will report information back to APM.

In terms of pricing, we know the per-host pricing is challenging for our users, and we're in the process of making that better. that's all I can say about that right now, but stay tuned.

Natasha: Maybe one extra thing I'd chime in to say is, we do understand the concept of a host has changed, so if you're in a situation where you have one host, but 100 Docker containers, you're like, "I'm not going to get charged for 100 hosts, I'm only running on one host." Just call us up, we'll talk to a person and work it out. Don't be scared off by the rules looking a certain way. That's my two cents there.

Tori: Exactly. Thank you, that was well put. I don't know if, maybe I'll read it, and then you'll handle it, Natasha. How do you handle input output of your Docker container on your infrastructure hosts?

Natasha: Great question. What we do, basically, the thing I'd like to say is that for I/O, basically writing to your local file system has gone the way of free checked baggage on airlines. It's just something you don't really get anymore, unless you pay extra for it.

Basically, developers have to assume that you can't really write to files. If your application depends on dumping a 500 gigabyte file, for example, in a batch job, once an hour, or once a day, that's something you've got to call and talk to them about now, because you can't do that in a Docker container by default.

Basically, your application either needs to persist I/O to an external source like an S3 bucket, or you've got to talk to your ops team about mounting an external volume to your Docker container in a special circumstance. Preferably, you just shuffle everything off the container.

For example, for logging, you use syslog, and nothing gets logged to the containers. Everything immediately, as it happens, gets streamed off to a centralized logging system.

Tori: Great, thank you. I love the baggage analogy. How do you combine Docker with something like OpenStack or CloudStack, or is Docker a replacement?

Natasha: That is a good question. Honestly, I can't. I'm not as well attuned to, one of the things that Tori mentioned in her Roundup slide is you do have to stay aware of what's going on in the community, because the platforms are all being invented as we speak, and so I have to admit, I've done that in a limited way.

I'm not familiar enough with OpenStack or CloudStack to be able to answer that question, I'm sorry to say, but I'd be happy to ask around and get back to the question answer offline.

Tori: Great. Do we have any more questions?

Natasha: Can we share the name of the register software in an email after the call? I just saw that one popped up. The registry software that we have is called Dogestry, I'll just go ahead and spell it out. It's like doge, the meme, D-O-G-E-S-T-R-Y. I think it's like a registry, but "Doge, such storage, very wow." That's a sort of joke.

It's up there, newrelic/dogestry is the repo, and it gives instructions for here's how you set it up on top of an S3 bucket, so that you can roll your own registry. It is the poor person's registry, the registries out there like Docker Hub and key.io have a lot of nice user facing functionality that Dogestry definitely does not have.

At the same time, you're able to take permission. We were trying to solve the specific problem of permission and, who can access what. It's good for that, and it's good for if you want to build it yourself.

Tori: Great.

Natasha: Another question?

Tori: Another techie question.

Natasha: Do we use another container to hold technology, or just Docker? At this time, just Docker. Because everything is being invented as we speak, you have to stay situationally aware of what other technologies are out there. We do try to keep abreast of other things that are breaking in the area, but right now, very much put our eggs in the Docker basket.

Moderator: Hello, everyone. Thank you for joining today's webinar, Scaling with Docker: New Relic's Containerisation Journey. You'll hear from Tori Wieldt, developer/advocate at New Relic.

Before we get started, just want to go over a few housekeeping items with you guys. Today's webinar will be recorded. A recording will be sent out to you shortly after the presentation.

There will be time reserved at the end for question and answers, so please be sure to place any questions you might have in the Question panel of the GoToWebinar control panel. Now, I'll hand if off to Tori.

Tori Wieldt: Hello, everyone. Good morning, afternoon, wherever you are. I'm delighted you joined us. I wanted to share a little bit about what we've learned at New Relic.

I do have to start with the obligatory Safe Harbor slide. I did put a typo in there. I'm challenging you to read this entire slide and find the typo. All right, just kidding. What it pretty much means is, don't make decisions based on what I've told you, although I'm going to give you some really great information, so maybe you should.

I wanted to share with you our journey to containerization. What we learned along the way, how we went from a monolith to a modern, scalable software stack, and hopefully give you some lessons and some insights into the things we've learned, and help you along the way on your journey.

I'll share what we've liked about Docker, what's worked for us really well. I will answer the controversial question, "Is Docker ready for production?" I'll tell you some of the challenges and limitations we've encountered along the way. How to look at what's happening inside your container, this is key.

You've got to know what's going on with your containers. How do you maintain your success? You've moved to a services architecture, how are you going to have the most success using Docker in containers? For those of you that don't know us, New Relic is a cloud-based software analytics company. We give visibility to our customers, into how their applications and their business is running.

We do this through a variety of agents, which are installed on the applications, and people use it to monitor their services in the cloud. All this comes together in the New Relic cluster, where we collect all that data, we process it, we store it, and we make it available for people to query in a fast, ad hoc manner using our Web and mobile interfaces.

The bottom line here, two bottom lines is, one, our application in the cloud is our product, so if we're not up, we have nothing to offer our customer. Also, we're dealing with lots and lots of queries a day, so that is a T for trillion. Roughly, and this may be updated, but we deal with over 100 trillion queries a day.

What was our journey like, what can we learn, and what can we share with you? We started out with the classic Monolith. It was a Ruby app, it was a single application, it ran our entire business. It contained the agent, the data collection timeline, and the Web interface. Back in ancient times, when it was divided out into the duo list.

This is with a Ruby on Rails application, which was the user interface, and then the back end was a Java collection pipeline. As our customer base grew, and our feature set grew, and our company grew, we started to have problems. Problems which are probably familiar to many of you.

As our success continued to grow, our problems continued to grow. How did we think about solving that? We moved to a services architecture. We looked in and decided we're going to have people create services, and we thought, "Oh, we might have dozens, or maybe at the top, 100."

The reality is, we're running 200 plus services, and we needed a way to deploy it faster, so we chose Docker. Keep in mind, we chose Docker in January of 2014. Those were the early days, and perhaps that was a little ambitious, but Docker had other characteristics we wanted.

It isolated the experience of developing applications from the maintenance of that service. It provided great efficiency, and good uptime. We looked at our office people and said, "Hey, instead of running two or three services, you have to run 200."

Docker provided a more standardized interface to let them do that. Here's how deployments worked before and after Docker. This is the generic view, it's not 100 percent specific to New Relic, but it'll give you a sense of the change that we went through. It's hard and expensive to get communications and processes right between teams of people, even in small organizations.

Detailed communication has to go between the development team and the operations team. You can see on the left, the development team would request a resource. The ops team would provision the resource. That would often take a long time. Then the development team would script the resource, and you would tweak the deployment. Of course, you had loop through this several times in terms of fixing problems.

You are probably all familiar with dependency hell, knowing what rev, what patch of your database, of your OS, of all the libraries that your app is using can be really challenging. With Docker, we're able to reduce that to a lot fewer steps. I don't want to create the illusion that there is no need for an operations team, and they don't have anything to do.

The graph on the right is pretty simplified. They still have to work on provisioning, and working on optimizing the infrastructure, and that's a key piece that we need to have done, but a tool that reduces the complexity of the communication, while aiding in the production of more robust software was a big win for us. That's exactly what we've found with Docker.

Now, Docker's no panacea, and implementing Docker takes work, but it's a good approach to solving some of the real-world organizational problems we had, and it helps us ship software faster. This is key. Designing a well-designed Docker workflow can lead to happier technical teams, and real money for our bottom line, which has been definitely our experience.

How do we use Docker? We use Docker for almost everything that New Relic serves up. There's some specific high throughput services that we don't provide on Docker, but for the most part, our whole stack is containerized. Here's a good example.

We have an application called New Relics Synthetics. We like to call it a crash test dummy for your software. What Synthetics does is, based on Selenium, and what it does is it can ping your application from all over the world. It can also work on simulating a login to your application, or somebody checking out something from the car.

Anything that's key for your application to use, you can synthesize that, and have a script do it for you on a regular basis. This is a key way we use Docker, because each script that a user spins up runs in its own container. We use this both for performance, and security reasons. Performance-wise, containers spin up so quickly. That's important for us.

Security-wise, it provides a much smaller, what we call it is an attack vector. That container is up, does its job, and goes away. It's a lot harder for somebody to hack that. We've been using Docker for this, and what we've seen is a lot of people using Docker in the specific way. The container spins up, does a small job, and then goes away.

What we'd like to call that is the ephemeral infrastructure. In this slide, I'm generally talking about, New Relic is in an interesting position. We use Docker ourselves, and also our customers use Docker, so we're able to look at how our customers are using Docker. We do see that customers, yes, they use Docker as a VM.

Just replacing a VM, doing a listing shift, but also they start using Docker in the cloud, in the EC2 environment. There's also this short-lived containers. A container is a process, if you will, lambda-ish, where the container comes up, does its job, and goes away.

We've found that fully 25 percent of our customer's usage is containers that last for less than five minutes. This is interesting in terms of making decisions about your architecture and infrastructure for containers that only last a short amount of time. What have we found in Docker that we like? Why does New Relic love containers?

Our developers are productive much faster. There's less yak shading, more time spent coding. I have to be honest here. This did not happen overnight, and I think the developers would personally come and throttle me if I didn't explain that it was bumpy in the beginning, but now that we have better tools, better way to support the developers, it does get them up and running much more quickly.

We can get people provision in the matter of hours, rather than weeks, and that's really important. Also, as I showed the streamlined communications between teams, it means less handoff, less errors. This is very important in terms of the title information that has to go back and forth between teams.

If you can get that organized in a better way, that's very helpful, and we like containers because they're better than VMs. Containers launch in nanoseconds. They're much faster than VMs in terms of launching, and utilization is four to six times better than VMs.

Since a separate kernel doesn't load for each user session like it does on a VM, the overhead associated with multiple OSes don't happen in containers. This is why containers use much less memory, and CPU, than VMs running similar workloads. For example, you can't put 100 VMs on a Mac, but I have run 100 containers on it.

Docker allows for scale. The abstraction allows for scale. We're talking about scale in several different ways. The key way people think about scale is input. How many users do you have? What's the throughput of your app? How many times are you getting ping? In reality, there's different ways that you have scale, and you have to think about it.

One is the size of the app itself. Another way is the number of features on your app. What are you providing to your users? You have to think about scaling up that way, and scale in terms of number of engineers that are working on your application. How can you scale that up?

One is, you want to get them productive quickly, but in another way, you want to make sure that what they're working on when they introduced changes to their application, their service, they're not causing disruption throughout the entire system. Abstraction allows for scale.

Docker allows you to easily deploy new containers, programmatically. Also, our system is elastic. It can grow and shrink with the load, automatically. Different pieces of your system, as I mentioned, can be swapped out, scaled up without affecting the rest. We can put out releases without downtime.

Why do the ops people love containers? There's no more activating, kickstarting of pizza boxes. This is not only a Docker thing, this is thanks to the cloud, but ops people don't have to get in there and physically spin up a machine. You just script what you need, and build once, run anywhere works for infrastructure.

As one ops person told me, ops is now Costco instead of the corner store. They're providing a big chunk of ice instead of special snowflakes. Also, we do an exercise with disaster recovery. As I said, our application is available. It's in the cloud, and so if we don't have an application up, we don't have anything to offer our customers.

We've done disaster recovery exercises, where we build all the services from scratch. 80 percent of the containers took 20 percent of the time. Those were the standardized Docker image, I'm going to put it out there, and then the special snowflake containers, those 20 percent of the special snowflakes took 80 percent of the time.

That's classic, but Docker did allow for better scripting, and better rebuilt, and for the disaster that is not going to happen, but it's good to be ready for it. Docker provides much less fiddling with Puppet and Chef. Now, developers own those dependencies, so they are responsible for figuring out what their dependencies are for their particular service.

Another thing that's great about containers, both development and ops, you don't spend a lot of time troubleshooting your container. What you want to do is just replace it. I have a friend that says, "Treat them like diapers. If you're having a problem, throw them away. They're cheap to spin up another one."

Why do developers love containers? Well, the faster provisioning is an important piece, not only businesswise, but if you think about the human part of it, I worked on this new functionality, I want to get it out to users as soon as possible. And so, telling the developer, "Oh well, it's going to take two weeks," it can be done in minutes.

Developer enthusiasm is pretty cool. When they worked on the future, they can get it out, and they can start getting feedback on it almost immediately. Developers also control their runtime environment. You own exactly what the app gets. Tool redundancy is "easy." I put easy in quotes because it's not 100 percent out of the box, doesn't require any tools.

There does have to be op support around this, but it's easy to put your image out, put a second image right next to it, run exactly the same thing, and then best practice is strap it across hosts. You want to have four images of your service running, two per host. That way, that redundancy helps with availability, in case a container goes down.

Docker is very microservices friendly, and it helps with the way of looking at your container. You can start thinking about architecting one service per container, and that can be a best practice that you can use. It's a great way to start thinking about, "Oh, I have a login service? That goes in one container. I have another feature? That goes in another container."

Another thing we've seen with Docker is, because these containers can run independently of each other, then you can pick the best language for a job. Not everything has to be in Java, not everything has to be in Rails. You can use the best piece. We have pieces of our application that are written in Go. For that particular functionality, Go is the best language, and that's what we use to support it.

The controversial question, is Docker ready for production? People have been asking us, "How long has Docker been out? A couple years now?" Everybody wants to know. Well, it's a trick question, because we are using it in production. Ready or not, we've gotten a lot of functionality out of it.

We've gotten a lot of good things that are happening, but it is challenge. It is a dynamically moving technology that we're using. Docker is a moving target, and pretty much, if you think about Docker as a system API that's strapped to things in the Linux kernel, so Docker doesn't have complete control over what they're using. If there are issues with the Linux kernel, that has to be corrected.

I do have to say, Docker is doing a great job. With each release, it's getting better and better. We're seeing better tools, better security, things like that. It is improving. I would say that I would stay away from being on the cutting-edge. [laughs] Your mileage may vary, but I'll talk about that in a minute. Also, wow, "Is Docker native on Mac and Windows?"

According to the latest release, it is. There's been the transition from using boot to Docker, and now Docker Toolbox, and now it's native. That's going to put some bumps in the road. If you're developers, most of our developers are in Mac, so it's going to change the way they work with Docker. Be ready for change.

Also, one of the challenges with Docker is the docks are written for green field development. There's a great Docker "Hello World" tutorial out at docker.com. I'd recommend taking a look at it if you're totally new to Docker, but it doesn't provide information on how to take an existing app and Dockerize it.

Most of that image, now, is in trivial form. How to Dockerize your data center. I haven't seen anything yet. I'm sure there's probably some blogs out there, but there's nothing that's detailed that tells you how it's going to do that. As the ops people have told me, Docker is lots of pain, but we do get a lot of gain.

One of the biggest challenges is getting that workflow into everybody's lives. What are the challenges and limitations to Docker? As we said, it's not a panacea. It's not going to cure everything. Here's some of the specific challenges we encounter. There is a learning curve of, people are going to have to learn the new tools, go through dev and ops. We'll need to acquire new skill sets.

On the outside, you'll need a solid library of base images, a process for building and shipping Docker, and you do need good internal docs and support for developers. This is part of the bumpy road when we roll out Docker, and didn't provide tools or information. It was frustrating for the developers to use it.

On the developer side, you have to learn those tools that have been provided to you and Docker, and you have to take ownership of your app's dependencies, and you have to let go of certain things, like having access to a persistent file system. Logging your app, you can't just log your app inside the container. It's going to fill up the container.

You have to start thinking about your app in a little bit different way. Also, as I mentioned, legacy apps. We found that converting legacy apps to Docker is a pain point. It's great once you know all the dependencies on your app, and you could build containers repeatedly, but sussing out those dependencies can be a tough exercise.

Once your application is running in a container, there are things that you take for granted, that you have in your own environment, that don't exist in a container. For example, that you can write to a temp directory, that just doesn't exist in your container.

Also, multi-threaded code. You can use the Docker scale command to easily create additional images of your additional containers of your images, but that doesn't make your code multi-threaded. You have to code your app to be truly redundant. Also, there is a noisy neighbor problem.

Containers are isolated from each other, but it's probably more limited than you might expect. While you can put limits on resources, the default container config has been sharing CPU and memory on the host. This means that unless you constrain them, containers will compete for resources on your production machine.

This is one of the gotchas that we encountered, is we did not put memory limits on our containers, so as the best practice, you absolutely just need to put some sort of memory limit, so it doesn't run away and cause problems for all the containers on that same host. There are issues with security on Docker.

Our ops people use base images to enforce some best practices around security, like preventing anyone from logging into a container is used. Patch management is also an issue. When we discover a vulnerability, we have to upgrade our whole environment quickly. It's obvious you have to patch the Docker host, right?

You have to update your Docker files for future revs of that host, and then you have to remember that developers are also running that whole environment on their laptops. You have to get them to pull the updated base images onto their machines, and it's a new habit that you have to build into your organization.

The limitations of Docker. Docker is great, it does a lot of things, but it is a dessert topping, and a floor wax. It doesn't take care of everything for you. What's your host? That's a decision you have to make for yourself. Docker is not the cloud. You can run Docker almost anywhere, and that's your decision about where you're going to run it.

You have to choose your cloud platform, if you're going to be in the cloud, and figure out what works best for you. Docker can solve a lot of things, but sometimes you need a specific tool with more in-depth features than Docker can offer. Docker can significantly improve your organization's ability to manage applications and your dependencies, but it does not directly replace traditional configuration management.

We still use Puppet. We also use Vagrant to support testing on images that match our production environment. Docker by itself can't automate your deployment process. We've deployed hundreds of times a day, and we started using Docker early on, and there were hardly any tools.

We built a tool that's called Centurion. It's a Docker deployment school, and with that tool, the build shifts containers to the Docker Registry, and then Centurion sends containers out to the Docker fleet. What it does is, it uses your Docker server form as a deployment target, so it handles things like volume mounts and port mapping.

We have open sourced Centurion. We're sharing it with the community. It's out on Github, under New Relics Centurion. Please take a look at it. If it helps you, you're welcome to use it. Being an open source project, we'd love for you to contribute back, if there's some code or some feature you'd like to improve on it.

How do you know what's happening with your containers? You have this great new delivery system, where you're delivering quite quickly. You have this ephemeral infrastructure. How do you know what the heck's going on? At the most basic level, there's something called the Docker Steps command.

This is a command line tool, and it gives you the basic resource consumption of your containers, and it'll tell you the names of the running containers. It'll present the CPU utilization, the memory, the total memory. Note that if you do not have limited memory for your containers, this will post the total available memory of your host, and that's not the total available memory to your container.

That doesn't mean that it has that access, but you, listening to me, make sure that you're already constraining the memory of your container, so this won't be an issue for you. In addition, you can see the total data sent and received over the network by the container. The step command is OK. You can also use the Docker API.

It's very similar to Docker Steps, and it can give you a live stream of what's going on with the CPU memory, and it provides a little more detailed steps than the steps command itself. How can you attach to a running container and see what's going on inside of it? These two commands are used by our ops people.

The first command, obviously you have to plug in your Docker host, and your port number, and the name of your app, but this graph command will give you, returned back a container ID that you can then use in the next command. That will connect you to your container, then you're running a bash shell inside the container.

That way, you can just troubleshoot in a way that you usually do, but we're running Docker at scale, so we don't want to look at individual containers. We don't want to spend time looking, tweaking around with individual ones unless we absolutely have to.

This is where I want to show you what New Relic does with Docker containers, and what information it can provide you. To provide some sort of contacts, our basic product is called APM, application performance management. This is a screenshot of the overview screen.

It gives you basic information about what's going on with your app. We want to look at your app, and see what's going on. If you scroll down on this particular screen, the overview screen, you're going to get a sense. You're going to get a table like this, that's going to tell you what's going on with your containers.

It's broken out by server, and lists each app instance in the cute little, I don't know, does that look like a Lego? The Lego icon there provides for you. It shows you that it's an individual container, and it's group by server. I've looked at an overview of my app, I see a server that I want to drill down on, and I want to get some more information on.

Each one of those is a clickable link, so if we go and look at our Docker form, it takes us over to the New Relic server product. This gives you basic information, similar to Docker Step. It gives you CPU usage, what's going on with physical memory, load average. This is the overview for a particular server.

This is interesting, but I don't think it's valuable. We believe you should focus on what's going on with your applications, instead of a particular server. This is what we provide in the server Dockers tab. It's broken out, it's rolled up by images. This shows us by CPU, the largest consumers of CPUs.

It's just a stack rank. This is what you care about. You can see the top list here. This image is running in two different containers. We combined it, and the CPU consumption is seven percent, nearly eight percent. The memory is around 30 gigabytes.

This is the detailed information, apps or services are roughly equivalent to images, so this is a great roll-up. This is a quick way to give an overview. We've used this a lot. It really minimizes the finger-pointing. If you're introducing a new service out on a container, and people are like, "Oh, you've just destroyed my system, or my host," this is a really good way to take a look at it and go, "Is what I introduced a problem? Perhaps it is not."

I hope that was helpful, giving you an idea of what our journey was, how to look at what's going on with containers. How do you maintain your success? What are some of the key takeaways we provide for you? One, I'd say make sure you start simple. Don't start with your most intense, high throughput app.

A great way to start is if you're adding new functionality to your application. It's a great way to break apart your monolith, use it as a micro service, and you can use that particular service in a container. Also, we found that a lot of the benefits we got from Docker came from organizing around the technology.

As I described, our whole workflow has changed. The way we interact, the hand-off between dev and ops has improved, and so we've gotten a lot of benefits out of looking at one technology, everybody across the company using it, and centralizing on that.

Of course, it's key to keep a good watch. Monitor what's going on. It's very important to look at what's going on with your application. You can do infrastructure in, or infrastructure up, but we think it's really important to know what the application is doing, therefore what is your user experience?

Nothing lasts forever. You have to architect for the ephemeral. Remember, I said that 25 percent of our users, their Docker containers last less than five minutes. What does that look like for you? How do you offer services where you can spin off a Docker container, and it goes away?

Also, nothing lasts forever. Docker's great, Docker's going to change a lot. Docker's going to get modified, and it may be replaced by another technology. That's the joy we get of working in a technology field. Things are always changing, so don't get too attached to it. It's very helpful, we've gotten a lot of benefits out of it, but keep in mind that there will be a new technology coming down the road.

Thank you for joining me. Here's some resources in terms of learning more about Docker. Docker does provide a nice "Hello World" tutorial that you can go through. We have two engineers who wrote the book, "Docker and Production," and that's been very helpful for people to take a look at.

I really appreciate the book because it's a nice look at the theoretical part of Docker, but it does offer some cookbook instructions on how to work on Docker, and scale it up. If you go to newrelic.com/docker, you can get the first three chapters of the ebook for free, so please take advantage of that.

There's a nice Docker tutorial, Docker for Java, for Java developers, developed by Arun Gupta, who's over at Couchbase. He was able to put this tutorial together, and share it with the community. It's out on GitHub, take a look. It walks you through the basics of Docker, how to work through multiple hosts, and also the tooling that's available on Eclipse where you can work with Docker directly through your IDE.

Thanks. I'm really glad you attended. I will be posting these slides to slide share, and you will get a copy of, you will get an email that tells you details. If you have any questions now, we'd love to hear from you, or also you can reach out to me on twitter. My twitter handle is my name, @toriwieldt.

Thank you. What questions do people have? Let's take a look. The first question is where do you put the configured information for the deployment information? That's outside of the container, in your Docker file. That's where you put it.

Natasha Litt: Hey everybody, I'm also a colleague of Tori's, Natasha Litt. I have some flight hours, both on the operations side, and on the development side with Docker. I'm joining in to help answer any technical questions. Tori, mind if I chime in on this one?

Tori: Absolutely.

Natasha: Great. The configuration information for each deployment environment goes in, definitely the Docker file has some configuration specifics for your app, but the way we try to approach it is basically that your runtime app equals your built container, plus a bunch of environment variables that you feed into it.

For example, like in staging, maybe you're talking to a staging database, and then in correction, you're talking to the big iron database. What we'll do is we'll boot up the Docker containers on our staging environment by feeding them environment variables that point the app to the staging database.

Whereas, the same container is fed a different set of environment variables in production, and that makes you talk to production instead. The open source tool that we wrote, Centurion, makes it really easy to basically create a little portfolio of environment variables that get fed to your container, in different environments.

That's in our own ecosystem, we use Centurion configuration files, rig files to store the configuration information. That makes it really easy to organize very large sets of, where we've got hundreds of repos, and we can still know where everybody's staging and production environment variables go.

That ties back into what Tori was saying about, the work it takes to Dockerize your app. developers have to take the time to find every hard-coded reference in their code, and extract it out into the environment so that it can be controlled from the environment, so that the image can run in production, or in staging either way, just depending on changing the environment variables.

Tori: Wow, great. It's great to have the pros who have been in the trenches to help out with that, thank you. Let's see, Chris has a question. I presume there some agent to be installed for APM. Is there a Docker container for that? What you do with the agent as you install it on your application, on your running app, that's how you get the APM information.

Then you have to do some sort of tweaking on the server side, and that's in our documentation. That way, servers talk to APMs, and that's how you can get the clean roll up information on Docker. Oh, Natasha, I'm glad you're here. Next question. If you have a private Docker registry, what does your naming tagging strategy look like for images?

Natasha: What a great question. We have a sort of private public Docker registry. Because the Docker registries out there, Docker Hub and key.io and that sort of thing, we like them, but for security reasons, we wanted to have very fine-grained controls over who could access what.

We created our own, we basically wrote some open-source software, it's called Dogestry. It turns an Amazon S3 bucket into a registry. We use that as our private Docker registry, except that the software we used to privatize it is public, so you could use it yourself, if you'd like.

What does the main tagging strategy look like for Image? Great question. What we do is, every time you build a Docker image that is a release candidate, you tag it, and you push that tag to the repo, or to the Docker registry. Then, whatever is the top of the tree you tag latest.

That way, you always know what's running on, what is the next thing to deploy. It's the latest one, and then if you need to revert to a certain shot, what we do is we tag each release candidate with the SHA of the last commit. You can easily look at your GitHub repo and say, "Oh, I need to revert to where the last commit was A3095," and so then you can just push the Docker image A3095, and make that the latest one. Now, you've rolled back to that state.

Tori: Excellent, thank you. We've Got another great question here. Robert asks, did we do testing to measure the overhead delay caused by monitoring apps and containers? I can take that. When we measure to check this out, overhead runs between two and three percent.

Next question. Since billing is per host, does New Relic know if multiple Docker containers were used on a single host? You need a New Relic agent on the host, and scripts agents within the containers. As we discussed before, the agent sits in your app, so if you're running a Java app, you just download the new Relic agent, deploy it, and then it really doesn't matter where it's running, it will report information back to APM.

In terms of pricing, we know the per-host pricing is challenging for our users, and we're in the process of making that better. that's all I can say about that right now, but stay tuned.

Natasha: Maybe one extra thing I'd chime in to say is, we do understand the concept of a host has changed, so if you're in a situation where you have one host, but 100 Docker containers, you're like, "I'm not going to get charged for 100 hosts, I'm only running on one host." Just call us up, we'll talk to a person and work it out. Don't be scared off by the rules looking a certain way. That's my two cents there.

Tori: Exactly. Thank you, that was well put. I don't know if, maybe I'll read it, and then you'll handle it, Natasha. How do you handle input output of your Docker container on your infrastructure hosts?

Natasha: Great question. What we do, basically, the thing I'd like to say is that for I/O, basically writing to your local file system has gone the way of free checked baggage on airlines. It's just something you don't really get anymore, unless you pay extra for it.

Basically, developers have to assume that you can't really write to files. If your application depends on dumping a 500 gigabyte file, for example, in a batch job, once an hour, or once a day, that's something you've got to call and talk to them about now, because you can't do that in a Docker container by default.

Basically, your application either needs to persist I/O to an external source like an S3 bucket, or you've got to talk to your ops team about mounting an external volume to your Docker container in a special circumstance. Preferably, you just shuffle everything off the container.

For example, for logging, you use syslog, and nothing gets logged to the containers. Everything immediately, as it happens, gets streamed off to a centralized logging system.

Tori: Great, thank you. I love the baggage analogy. How do you combine Docker with something like OpenStack or CloudStack, or is Docker a replacement?

Natasha: That is a good question. Honestly, I can't. I'm not as well attuned to, one of the things that Tori mentioned in her Roundup slide is you do have to stay aware of what's going on in the community, because the platforms are all being invented as we speak, and so I have to admit, I've done that in a limited way.

I'm not familiar enough with OpenStack or CloudStack to be able to answer that question, I'm sorry to say, but I'd be happy to ask around and get back to the question answer offline.

Tori: Great. Do we have any more questions?

Natasha: Can we share the name of the register software in an email after the call? I just saw that one popped up. The registry software that we have is called Dogestry, I'll just go ahead and spell it out. It's like doge, the meme, D-O-G-E-S-T-R-Y. I think it's like a registry, but "Doge, such storage, very wow." That's a sort of joke.

It's up there, newrelic/dogestry is the repo, and it gives instructions for here's how you set it up on top of an S3 bucket, so that you can roll your own registry. It is the poor person's registry, the registries out there like Docker Hub and key.io have a lot of nice user facing functionality that Dogestry definitely does not have.

At the same time, you're able to take permission. We were trying to solve the specific problem of permission and, who can access what. It's good for that, and it's good for if you want to build it yourself.

Tori: Great.

Natasha: Another question?

Tori: Another techie question.

Natasha: Do we use another container to hold technology, or just Docker? At this time, just Docker. Because everything is being invented as we speak, you have to stay situationally aware of what other technologies are out there. We do try to keep abreast of other things that are breaking in the area, but right now, very much put our eggs in the Docker basket.

I know that the ops team, there is a crew of people who are always checking out new technologies, and assessing to see, "Hey, it'd be useful to incorporated this." They're always poking at stuff like that, so I only hear about it if they find it interesting enough that it percolates through the engineering grape vine that way.

Moderator: OK, thanks. Those are all the questions we have today. Thanks, Tori and Natasha, for your time in doing this webinar. Just a quick reminder, a recording will be sent out to you guys shortly. In the meantime, don't forget to check out newrelic.com/docker. If you have any further questions, feel free to email Tori or tweet her. Thanks again for joining us, and have a great day.

Tori: Thank you. Bye-bye.

Natasha: Thanks so much.

Moderator: OK, thanks. Those are all the questions we have today. Thanks, Tori and Natasha, for your time in doing this webinar. Just a quick reminder, a recording will be sent out to you guys shortly. In the meantime, don't forget to check out newrelic.com/docker. If you have any further questions, feel free to email Tori or tweet her. Thanks again for joining us, and have a great day.

Tori: Thank you. Bye-bye.

Natasha: Thanks so much.

Read more