There’s a wealth of resources available on starting your project with Docker. However, if you’ve been in the game long enough, you’re facing a different problem: How do I migrate the codebase I’ve been developing for years to that new hotness? And does it even make sense to do that?
Perhaps our experiences at New Relic will help put the issue in perspective. Over the course of a few weeks, our visualization team has adopted Docker across the development pipeline. We embarked on this journey filled both with hope and fear. We heard all about how Docker improves relations on the developers-sysadmins axis, how it makes isolating and reproducing bugs trivial, and how it pretty much embodies the future of everything. On the other hand, having never used it before, we were understandably suspicious about the hype. As Docker newbies, we also faced a lot of problems, and while looking for solutions we ended up with a set of “less than worst” practices that might benefit teams in a similar position.
For example, when we set out to Dockerize an upcoming visualizations project, we defined a set of goals:
- Deploy all parts of the project as Docker containers
- Use New Relic’s open source Centurion tool for deploys (see How New Relic Used Docker to Solve Thorny Deployment Issues)
- Have images built automatically by Jenkins
- Use the same images for development and for production
What follows is an account of how the visualization commandos fought, bled, and finally waved their victorious flag over Dockerland.
Chapter 1: Deciding image layout
Our project is made out of at least eight microservices working independently together. We knew each of them would have to run as a separate container, which in Dockerspeak means a running instance of an image. What’s an image then? It’s basically a snapshot of a filesystem along with some metadata telling Docker what program to run when we ask it to create a container. (More about images and containers and other wonderfully confusing terms can be found in Docker’s documentation.) What we needed to decide was if we wanted one image per service, a single configurable master image, or something in between.
Our codebase is split across two repositories: one containing backend services and another with the UI code. We ended up building two images per repo, a base image and the main image.
The base image is the runtime environment: directory layout, OS users, packages, and so on. The main image builds on top of the base and just installs the app, nothing more.
This mirrors the repository layout nicely and avoids having to juggle too many very similar images. The rationale for the base/main split was primarily to keep build times low by reusing base images between consecutive builds.
An environment variable is used at runtime to decide which service will run inside the container. This way, all backend services use the main backend image and both the Web UI and the Web task queue use the frontend image.
The main images are the ones deployed in production.
Chapter 2: A maintainable way of building images
Building the actual image is a fair bit of work. We create an OS user for the project, install runtime requirements, write out config files, and so on. To maintain all that in a readable and version-controlled way, we use Ansible for the actual image build.
Ansible is a config management tool, like Puppet or Chef, but it’s zero-bootstrap (you don’t need to install anything except Python on the target machine) and written in Python (so we can hack on it if need be).
Ansible allows using Jinja2 templates for config files, overridable variables, and adds some structure to your image provisioning. It enabled us to escape from the Glorified Shell Script (aka Dockerfile) approach to something much more maintainable. As a result, our Dockerfiles for base and main images are very simple. By the way, did you notice that we’re using supervisord? The reason for that is, surprisingly, logging, and will become obvious later.
Some of our services need a lot of configuration. And by a lot, I mean hundreds of values, all of which need to be passed into the Docker containers via environment variables. We deploy those services on dozens of hosts and things would quickly degenerate into a mess if it wasn’t for Centurion, one of New Relic’s open source projects. Centurion is a tool to centrally manage configurations for fleets of Docker services and it has been instrumental in formalizing the handoff between developers and system administrators. It has had a lot of success both internally at New Relic and in the wider community.
Chapter 3: Build automation
Time to make Jenkins earn his salary. The way we set up Docker image builds in Jenkins is:
- A git push builds a new package with the code.
- A VERSION identifier is generated as
- The package gets uploaded to an internal repository.
- The base is built and tagged with VERSION.
- The main image is built by installing the freshly created package into the base image.
- Both base and main images are pushed to a private Docker registry.
In our case, running Docker builds for the base images takes around eight minutes. In today’s attention-deficit disorder days, that’s plenty long, but here is where the Docker cache can help. Thanks to how our Dockerfile is laid out, we invalidate the cache only if we changed something in Ansible, or updated the list of packages on which the app depends. Otherwise, Docker reuses the previously built image.
The result is that most of the time, the base image is reused and the whole Docker build is just installing a new package into an already existing image. Builds of the main image run with caching disabled, since we always know there’s a new package available—Jenkins just built one!
Interlude: Running tests with Docker images
What’s the fun of using Docker if you’re not leveraging it to have the same environment from development through tests up until production? Our build job leaves a fresh production image on the build server and the downstream test job launches tests inside that container, using environment variables to make the tests use a local database, queue server, and so on.
The test output folder is bind-mounted into the host, so we can, for instance, pick up the coverage report afterwards.
Chapter 4: Logging in Containerworld
We like our logs. For debugging obscure issues, they’re something very nice to have. But here we are, running inside ephemeral containers, with nowhere to store them, sprawled across several hosts. The answer was centralized remote logging.
All of our code is using syslog, which gives us the flexibility to configure actual logging separately from the application. We run a syslog daemon alongside actual app code in every container, configured to forward logs to a central logging server.
By the way, why not just write logs to stdout, like some tutorials suggest? The issue is that our Web UI containers run several things that can log:
- Nginx producing your everyday httpd logs
- The WSGI daemon
- The application itself
We didn’t want to have those things interspersed. Both Nginx and uWSGI can be configured to log to syslog, so syslog it was.
That brings about two more problems (they just keep on appearing, don’t they?). First, what happens to logs generated before rsyslogd starts? We solved that one by wrapping service startup scripts in a simple loop that waits until the /dev/log socket becomes available before starting the real deal.
The second one was how to keep logs emitted during local development from being sent to the central logging server? We fixed that through (as you might have guessed) environment variables. If the log server address is not passed into the container, it reverts to logging locally. The log directory is a bind-mounted directory on the developer’s machine, so we can follow all the logs locally while working on the code.
Interlude: Monitoring Docker
As the saying goes, it’s not up if it’s not monitored. You don’t want to just start up a couple dozen containers and hope for the best, which is why if you’re running Docker, you should look into New Relic’s Docker monitoring capabilities.
Outro: Was it worth it?
In retrospect, yes, definitely. Unifying the deployment pipeline across local dev, staging, and production meant less problems reproducing production issues locally. The handoff and deployment story also dramatically improved. But you probably already knew that, since people have been repeating those things about Docker since the very beginning.
There’s an additional advantage to containerizing your application. It forces you to think hard about configuration, limiting the amount of mutable state inside your environment and your ability to scale horizontally. An exercise in switching to container-based deploys is actually an exercise in good engineering practices and any extra work required by Docker pays off by making your codebase better factored and less brittle. Onwards to excellence, riding the blue container whale!
Want to learn more about using Docker in the real world? Get a free sneak peek at Docker Up and Running, by Sean P. Kane and Karl Matthias.