Docker and Kubernetes
Published 2020 Mar 02 @ 21:53
It’s possible to now have Docker Desktop run a single-node Kubernetes cluster, which is wonderful for anyone that needs to emulate the Kubernetes environment for their development purposes. After all, if you’re building an application that will be deployed via Kubernetes, it is helpful to have your development environment mirror your production environment when testing certain features. However, it’s not all sunshine and roses…
First, Docker Compose
I am a huge fan of Docker Compose. I advocate its use whole-heartedly for local development. My career has primarily been backend development, meaning that I have to deal with at least the application and a database, sometimes a caching service. And that’s just the baseline. There are times when I’ve also needed to run elasticsearch, Kafka, and others.
The first big step toward having a simple and consistent development experience was Docker’s ability to easily “containerize” an application and run it locally. The next big step was Docker Compose and its ability to run multiple related images as a set of services that composed a full application stack.
I won’t go in-depth about Docker Compose, but I will just say that it has sped up the onboarding and development processes for every team I’ve been on that has used it. I can’t imagine life without something like Docker Compose now that I’ve been spoiled by it.
However, Docker Compose is primarily meant as a developer tool, not necessarily to spin up environments that mirror production, meaning there are serious limits on what you can accomplish through its configuration. Moreso with Kubernetes, which has a very deep set of customization options, almost none of which are exposed by Docker Compose.
Since I needed to (a) deploy in a Kubernetes environment and (b) have access to Kubernetes’ Downward API, I began searching for Docker Compose alternaties…
How about Docker Stack
At this point you might stumble across Docker Stack, like I did. Although originally intended to be used with Docker Swarm it can also be used with Kubernetes. If you have the latest Docker Desktop installed then you can enable a single-node Kubernetes cluster just by clicking a checkbox in the settings menu.
In the case of Kubernetes, Docker Stack takes a Compose config and uses it to
generate a series of Kubernetes resources. What’s better, you can easily update
your running cluster by re-running
docker stack deploy. You can also easily
tear it all down with
docker stack rm. It really makes emulating a Kubernetes
deployment as simple as one can imagine. Better yet, it just works.
Unfortunately, it still uses a Docker Compose config, which means you still get
all the major restrictions that come with it. There are some additional
options that Stack accepts which are normally ignored by Compose, such as the
replicas option to deploy multiple pods of a service, but ultimately they are
So if Docker Compose and Docker Stack can’t help me, what’s next?
Creating Kubernetes resource YAMLs by hand is always an option, but it’s easy to mess that up, and I didn’t want to try and map my working Compose config to a working set of Kubernetes YAMLs if I didn’t need to.
Eventually I gave up trying to “run” my Compose config directly and instead focused on tools that could convert a Compose config into one or more Kubernetes YAMLs. That’s when I found Kompose.
At first glance, Kompose looks like a Kubernetes-specific version of Docker
Stack. You can
kompose up and
kompose down, passing in a Compose config
file, and it just works. However, you can also use the
command to turn that config file into Kubernetes YAMLs. Yatta!
There were only two minor problems for my particular use-case:
- Volumes that bind mount to the local file system instead create a persistent volume that does not point to the local file system.
- The “shared” network I had to avoid explicitly exposing ports to some of my services (e.g. my database) did not work as expected, meaning anything without explicit port exposure wasn’t accessible in the expected manner.
To solve the bind mounting issue, I simply deleted the persistent volume claim
resource and associated metadata, then I added
hostPath configuration to my
deployment resource. See the official
for details on how to configure that.
Instead of trying to solve the issue of a “shared” network, I simply exposed the
port of my database in my Compose file and re-ran
kompose convert. That
generated a service resource in addition to the deployment resource, which was
good enough for my use-case.
After that, I just needed to add the extra metadata to use the Downward API in my application deployment resource.
I will still use Docker Compose for a majority of my local development. It’s an extremely quick and simple method for getting a working environment running, in addition to creating local bind mounts so that you can edit code on your host machine and have the changes show up in your running container simultaneously.
However, I can now use Docker Stack and Kompose when I need to test my app in the context of Kubernetes. Specifically,
- Docker Stack is useful when I need to have my service run in Kubernetes without any Kubernetes-specific configuration options.
- Kompose is useful when I need to convert my existing Compose config into Kubernetes YAMLs so that I can extend them with specific configuration options.
I have another post about Using GitLab AutoDevOps to build and deploy my application. If you’ve seen that post, or already know what GitLab AutoDevOps is, you can skip to the next paragraph. In short: AutoDevOps uses a custom GitLab pipeline with a custom Helm Chart to test, package, and deploy an application to Kubernetes. Ultimately, making large changes to how steps in the pipeline are executed typically requires using a custom pipeline step or a custom Helm chart.
Because I did not wish to use (read: was not having success with using) a custom
Helm chart, I ended up finding a workaround that didn’t involve using the
Kubernetes Downward API. Because all containers have the same hostname as the
pods they run on, it’s possible to use the
hostname -i (or similar) command to
get pod’s IP address. I use that to set an environment variable on the command
line when executing the application start-up script.
Although I didn’t ultimately get around to using Downward API in production, I was able to test out the application clustering in Kubernetes that I was hoping to do. So it wasn’t a total loss. And I certainly learned more about Kubernetes resource configuration along the way.