Building Tiny, Reliable Docker Container Images
Building good, clean Docker container images is a bit of an art, and there is a lot of conflicting advice out there about how to do them properly. I’d like to share some thoughts gained from running Docker containers in production for two years at New Relic. Some of this is discussed in the O’Reilly book Docker: Up and Running that I co-wrote with Sean Kane. There are all kinds of best practices we could talk about. Here I’ll focus on a few best practices aimed at making things small and reliable.
At scale you’re going to be shipping around tens, hundreds, or thousands of copies of your image. Docker’s distribution mechanism means that each host will need to have a copy of all of the layers of your image locally. If your image is 800MB then you’re going to have 800MB of data to pull at least once. There are a few prongs of attack to get that size down to a minimum. Some of these intermesh nicely with also making your images more reliable, as you’ll see.
Use Standard Base Images
Savings: Depends. Big environments will save a lot, smaller ones won’t.
Build all of your production containers on top of a simple set of base images. If your base image is big (usually meaning it’s based on a full OS distro), then with this pattern you still have to pull all of those layers, but you don’t have to do it for every application on each deployment. Since the layers are shared between apps, the overhead is reduced. There are all kinds of commonality benefits to be had here, too, so it’s a good pattern even if you don’t need the space.
I recommend constructing an image hierarchy with a build job that rebuilds, re-tags, and re-pushes all the affected base images when any upstream node is changed in the tree. This is a really nice pattern that has worked out well in my experience. With this pattern of building all the affected nodes in the tree whenever a change is pushed, anyone who will build an application image derived from one of those base layers will get the newest version of the base they depended on, even if it is an upstream image that changes. It also means you detect breaking changes to downstream images immediately, not when an application build fails down the road.
Here’s a linear example:
OS base -> Webapp base -> Ruby webapp -> Your Application
The behavior you want is to make sure that if security updates were applied to the “OS base” image, that your “Webapp base”, and “Ruby webapp” are automatically rebuilt, re-pushed, and re-tagged. Or imagine that the Nginx config in “Webapp base” was just improved. You want all future builds of “Your Application” to pick them up even though it’s built from “Ruby webapp” and not “Webapp base” directly.
Don’t Ship a Whole Linux Distribution
If you’re building and deploying Docker containers
FROM ubuntu or
centos and the like, then you may be causing yourself a lot of unecessary
pain. In some cases this is the right thing to do. But for many applications
you can get away with much less. I’ll talk about the MVP here in the next
section. But let’s assume that you need a shell and maybe some other tools to
bootstrap your application. That’s why you’re using one of the big distro base
images. The good news is that there are great alternatives out there. I won’t
go into all of them, I’ll just tell you about my favorite: Alpine Linux. This
is a tiny distribution, aimed at embedded systems and other small
installations. It’s perfect for containers because it has a full package
manager, a lot of available packages, and there is a good Docker base
image being maintained.
So, next time try
FROM gliderlabs/alpine:3.3 and see if it works for you.
Statically Link Your Applications
Savings: Potentially huge. In many cases no benefit.
If you can get away with it, you can ship the most minimal application of all:
your application binary and assorted supporting files. Rather than even using
Alpine Linux as your base, here you just declare
FROM scratch. This is a
great way to ship Go applications, or Rust, or C, or other compiled languages
where the application artifact can be a statically linked binary. If you’re
running a JVM, or a Python, Ruby, or Node app, then this is probably not a
solution for you. But if you can get away with it, there is basically 12KB of
overhead here on top of your application. That’s pretty minimal! Your
Dockerfile then shrinks to a few lines, with
FROM scratch and then adding
your application and its configs.
Process Management is Important
You need a program running at the top of your container that is meant to be run
as PID 1 on a Linux system. That process is usually SysV
init or Upstart, or
Systemd on the major distributions. You need something like that in your
container. Phusion wrote a good
post explaining why this is
important so I won’t rehash that here. Suffice it to say that you need a real
PID 1 process at the top of your container tree. But there are other reasons.
There is a very worthy goal in the Docker community of running as little in your container as possible. I applaud that. But reality is a harsh master and it turns out that in widespread production deployments lots of things go wrong no matter how well you build them. And as they get more fluid, with changes flying constantly, they break more often. The best solution here is to use a platform that schedules your containers to hosts and manages their life cycle. Examples are Mesos with Marathon, Kubernetes, Deis, and friends. Even so, it’s often the case that you need to run more than one process in your container and because of that you must have something that makes sure that dependency inside the container is maintained. Docker and outside schedulers can see the container but if one process in the container dies they may not notice. Don’t go overboard running lots of things in a container. Your container should do one thing. But doing one thing doesn’t mean you must have one process.
Enter the process manager. A lot of alternatives exist in this space. But you
want something that can act both as a real PID 1 and also does active process
management. Phusion recommends
runit which is a perfectly fine solution. If
you are running a big distro, you can easily use Upstart or
systemd. I have
supervisor here extensively and can recommend its robustness, but it
doesn’t handle the other duties of PID 1 (reaping children, signals, etc).
Also, a negative is that it needs a whole Python environment which can add
25-40MB to your container size.
My new favorite in this space is Skaware’s
S6, a not-that-well-known alternative that is
dead simple to use and configure. It’s minimalist but without giving up
ability. The binaries are statically linked, very small, and each do one thing
systemd). Installing it in your container image involves unpacking
a single tarball and dropping a config file.
One final point in this section. If you’re not running a distributed scheduler, you owe it to yourself to use a process manager, even if there’s only one other process in your container. Without something managing your application health, you’re relying on Docker to restart your dead container for you. My experience says that’s a much worse than 50-50 proposition. A process manager will make sure your container stays where you put it. Without one, you’re on an upside down roller coaster with no safety bar.
That’s some coverage of a few simple ideas that can have a big impact on the reliability and sustainability of your Docker images. Docker is a great tool but getting the platform right around is not trivial. This advice is based on two years of real production Docker use where we shipped 75+ application deployments on Docker per day. It should stand you in good stead.
comments powered by Disqus