New Relic Generates Real Data on Docker Container Adoption

Containers have enjoyed a tremendous amount of hype in recent months, but until now, data on patterns of container use in the real world has been limited to anecdotes and reports from individual companies, who could be outliers. We believe that's a big issue, because the question of how software teams are actually using containers is critical to the prioritization of container development in the open source container community and to the development and improvement of our products. Even more important long term, the rise of containers as a compute architecture could enable new application architectures with significant implications for software development and monitoring.

See also: New Relic’s Docker Monitoring Now Generally Available

Well, with Docker monitoring on New Relic, we now have some real data on key questions like:

• Is the container hype real? Is Docker maturing from proof-of-concept projects to production use?
• Are container-based applications being used by a broad swath of companies or is the technology limited to a narrow range of companies with unique software challenges?
• Does the lifespan of a container indicate a use of containers as lightweight servers (servers as pets) lightweight virtual machines (servers as cattle)? Or do container lifespans instead indicate net new application architectures indicated by very short container lifetimes? (servers as bacteria)?
• Which performance metrics are most important to surface to customers running Docker versus simply exposing all the metrics available to collect and analyze?  For example, if containers are being used as lightweight virtual machines, then we should monitor them as hosts. If containers are used not exclusively as VMs, but as ephemeral entities, then we should highlight performance metrics on the behavior of container types instead of the way we would traditionally monitor virtual machines.

As of June 15th, 2015, more than 300 New Relic customer accounts have activated the New Relic Docker monitoring beta, using an aggregate of 40,000 to 60,000 containers daily, and have collectively reported more than 2 million containers since the beginning of our private beta program in May. It’s still early, but we believe the data reflects that containers are being used in a growing number of production environments.

How about the lifespan question? The chart below shows a log scale of container lifespans by hour. For the containers that last for days or a week, the use case is likely medium lifespan servers (cattle!). Our suspicion is that the long-running servers may be proof-of-concept lightweight VMs that have been left running.

However, the dense gathering of containers over on the left is interesting, so let’s take a closer look at those below.

Here we see raw container volumes by minute for containers with lifespans of less than an hour. The large volume of containers (generated by multiple customers) lasting less than 5 minutes indicates the potential for net new application architectures using containers for periods of time far less than the amount of time typically needed to activate a virtual machine.

As noted, this is still early days. Docker usage patterns are not standard across the industry. People use Docker in unique and unexpected ways—something we discovered with this data—and there is no one single solution that will cater to the entire Docker market. The products we develop have provided visibility into containerized applications. We believe container technologies will continue to evolve the software development workflow and application architectures monitored by New Relic services. As always, we’d especially love to hear how you’re using containers to build new applications.

Note: Waseem Quraishi and Adam Larson contributed to this analysis.

Container image courtesy of Shutterstock.com.