Launch multi-node Kubernetes cluster locally in one minute, and more…

Note: For those who are interested in the video tutorial corresponding to this post, you can find them all on YouTube, or YouKu.

Motivation

To quickly deploy and launch Kubernetes cluster on your local machine is very useful in variant scenarios such as local testing or self-learning on Kubernetes deployment, and even the development of Kubernetes itself.

There are different ways to deploy Kubernetes cluster locally. For example, Minikube implements a local Kubernetes cluster on macOS, Linux, and Windows. Docker Desktop also has integrated Kubernetes on Mac and Windows since version 18.06. This makes it really convenient. However, they currently only support single node cluster. Single node cluster has its own limitation. For example, it cannot work with scheduler-related tasks or run tests that require multiple nodes.

Another disadvantage for some of these approaches is that you may not be able to specify a particular version of Kubernetes very easy. For example, my Docker Desktop uses Kubernetes v1.10.3. If you want to try some new features that are only available in newer or the latest version, you may have to seek for other solutions.

In this post, I will introduce a great tool, kubeadm-dind-cluster, that can deploy multi-node cluster on a single machine based on kubeadm and DIND(Docker in Docker). It can not only switch among different Kubernetes versions easily, but can also use the latest snapshot by building from source.

Moreover, to optimize the use of kubeadm-dind-cluster and make the bootstrap of Kubernetes cluster more efficient, I created a demo project, lab-k8s-playground. It is based on kubeadm-dind-cluster with some additional optimizations that I will describe later in this post. At the end of this post, you will see that we can launch a three-node Kubernetes cluster in less than one minute even without network connection!

kubeadm + DIND

Instead of using bare metal or virtual machine, kubeadm-dind-cluster uses Docker container to run cluster node. With that, all nodes can be deployed and run as containers on a single machine, typically your local machine. Application containers that previously run on Kubernetes nodes will in turn run in containerized nodes. That’s where DIND comes into play as it allows one Docker container to run in another Docker container!

This is not only valuable for local development and testing, but also for CI (Continuous Integration), because in many cases mordern CI environment is containerized itself too. To run application container in CI container, it requires DIND. When use kubeadm-dind-cluster, it can be run in CI environment without having issues with such kind of nested virtualization.

Similar Project

A similar project, nkube, starts Kubernetes clusters in Kubernetes (so-called nested Kubernetes). Essentially, it is also based on DIND, but requires a Kubernetes cluster (usually Minikube) to be run prior, then deploy the target Kubernetes on top of it. It is not very straightforward.

So, if you’re feeling yourself adventurous like me, you may give kubeadm-dind-cluster a try!

Basic Usage

It is easy to launch Kubernetes using kubeadm-dind-cluster. Just follow the README to download the pre-configured script here, e.g. the latest version at the time of this writing is dind-cluster-v1.14.sh. It maps to Kubernetes v1.14. Then run below command:

$ ./dind-cluster-v1.14.sh up

It will take a bit longer for the first launch because it requires to pull images and download dependencies from public network. Any other launch after then will be much faster, because it will create a snapshot locally after its first run so that can restore from snapshot quickly anytime later. When the launch is finished, by default it will bring up a three-node cluster, one master and two workers. This is configurable.

If you want to take down the cluster for some reason, use down command and this will keep the snapshot, so that can restore from it next time when you run up command:

$ ./dind-cluster-v1.14.sh down

For a clean launch without using snapshot, run clean instead of down. This will remove the snapshot.

Speed Up!

Using Private Registry

To restore from snapshot makes Kubernetes launch faster. It can be much faster for clean start if we use private Docker registries. As I mentioned earlier, kubeadm-dind-cluster pulls images from public Docker registries such as Google Container Registry and Docker Hub during the launch. The below list summarizes all required images from k8s.gcr.io in order to setup a Kubernetes cluster:

Images
k8s.gcr.io/pause
k8s.gcr.io/etcd
k8s.gcr.io/coredns
k8s.gcr.io/kubernetes-dashboard-amd64

If we copy these images to a private Docker registry that is co-located with the Kubernetes cluster on the same machine or in the same LAN, and have the cluster pull images from there, it will make the bootstrap dramatically faster!

This has been added into lab-k8s-playground, where you can find all private Docker registries configured in docker-compose.yml including k8s.gcr.io. Each registry itself is also a container that can be launched by Docker on the same machine where our Kubernetes cluster runs. All registries have a shared volume vol-registries.io to store images, as well as a shared bridge network net-registries.

For those who are interested in how to import upstream images from public Docker registries to a local private one, please check my other two posts: Docker Registry CLI Tutorial - Basic Use and Docker Registry CLI Tutorial - More Use. After the private registries are populated, we can launch them by simply running below command:

$ docker-compose up -d

Then we can run dind-cluster-wrapper.sh in lab-k8s-playground to launch the Kubernetes cluster. The shell script is actually a simple wrapper of dind-cluster-vx.xx.sh. It downloads specific version of dind-cluster-vx.xx.sh from GitHub repository to the current folder on your local machine if it is not existed. You can specify DIND_K8S_VERSION as the Kubernetes version and DIND_COMMIT as the corresponding Git commit id, or use the latest version by default.

Note: The kubeadm-dind-cluster Docker image required to launch the cluster is tagged by both DIND_K8S_VERSION and DIND_COMMIT. It is because there could be multiple tags for a specific Kubernetes version. You can find all tags with there commit ids here, then pick up one as needed.

Since it’s just a wrapper, you can use the same set of commands as you do when run dind-cluster-vx.xx.sh, e.g. up, down, clean.

Before run the shell script, we need to configure kubeadm-dind-cluster to recognize the network shared by the Docker registries so that kubeadm-dind-cluster can pull images from there. Also, if the registries are not TLS/SSL enabled, we need to let kubeadm-dind-cluster know. These can be done by exporting the below two environment variables:

$ export DIND_CUSTOM_NETWORK=net-registry
$ export DIND_INSECURE_REGISTRIES='["k8s.gcr.io", "gcr.io"]'

Run Offline

Since we now use private registries to pull images, and it runs on the same machine. We can even cut off the network connection during the cluster launch! The only thing you need to do prior that is to set the environment variable DIND_SKIP_PULL:

$ export DIND_SKIP_PULL=1

The kubeadm-dind-cluster image is pulled from Docker Hub. But it doesn’t have to be pulled all the time if we have it cached at local after first launch. This flag tells the shell script not to connect to Docker Hub to detect and download the image. With that, you can cut off your network connection, and see the bootstrap still runs perfect!

Skip Dashboard

kubeadm-dind-cluster deploys Kubernetes Dashboard by default. In some cases, this is not necessary. For example, we don’t need Web UI if we just focus on command-line interface kubectl. In such case, to skip Dashboard deployment during the launch will make the bootstrap even faster! This is also supported by specifying environment variable SKIP_DASHBOARD.

$ export SKIP_DASHBOARD=1

With all the optimizations above, a launch that restores from snapshot only takes 40+ seconds on my local machine! A clean start that pulls images from private registries and skips Dashboard deployment only takes 2 mins and a half! And, all can happen without network connection! That is amazing!

More Usages

There are some other usages that you may be interested.

Latest Dashboard

Sometimes we may need Dashboard. By default, kubeadm-dind-cluster deploys Dashboard v1.6 which is an old version. To use newer or the latest version, you can set the download URL for the Dashboard deployment YAML using environment variable DASHBOARD_URL.

$ export DASHBOARD_URL="https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml"

There’s one thing to note: since version 1.7, Kubernetes Dashboard enforces its security and can only be accessed over HTTPS by default. The access URL will be printed to the console at last when the cluster launch is finished. Make sure to pick up the right URL based on which version that you deploy and whether or not HTTPs is enabled. Here are sample URLs:

• If 1.6: http://127.0.0.1:33000/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy
• If 1.10 and HTTPs enabled: http://127.0.0.1:33000/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

Helm Installation

Helm is another great tool that people frequently use to deploy applications into Kubernetes cluster. I’m not going to describe how to install it here. You can find all the installation details here.

If helm CLI is installed, after kubeadm-dind-cluster brings up the cluster, you can run below command to install Tiller, the server portion of Helm, into your cluster:

$ helm init

Then run below command to verify the installation. You will see the Server version information after Tiller is installed:

$ helm version
Client: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}

Finally, don’t forget to create a snapshot by running snapshot command:

$ ./dind-cluster-wrapper.sh snapshot

So, next time when you bring up the cluster by kubeadm-dind-cluster, you will not have to reinstall it.

Use Different Versions

As I mentioned earlier, we can choose which Kubernetes version we use to launch. This is done by setting the environment variable DIND_K8S_VERSION, e.g. this is going to launch Kubernetes v1.12:

$ DIND_K8S_VERSION=v1.12 ./dind-cluster-wrapper.sh up

Launch from Source

We can even run the latest snapshot of Kubernetes by building from source. This is natively supported by kubeadm-dind-cluster and inherited by lab-k8s-playground. You just need to git clone Kubernetes source, set BUILD_KUBEADM and BUILD_HYPERKUBE to y:

$ export BUILD_KUBEADM=y
$ export BUILD_HYPERKUBE=y

Then run the wrapper from the root folder of Kubernetes source code.

$ cd kubernetes
$ ../lab-k8s-playground/dind-cluster-wrapper.sh up

The first run can be slow because it needs to build the base image and Kubernetes binaries. But subsequent runs are much faster.

(The end)

---

Have fun!

MorningSpace Lab