Building a Container Platform from Scratch (Part 1)

Published on the 17th of July 2024

I'm taking a small break from my regular KKnD 2 hacking to do something a little bit different. At the very least this will break up the front page of my site, which is currently entirely dedicated to the reverse engineering of obsucre file formats for a game released 26 years ago. Instead I want to tackle something closer to my day job. I'd like to build my own container platform, like Kubernetes, from scratch.

In case you're not familiar with Kubernetes, it is described as an open-source platform for "managing containerized workloads and services". It was originally developed by Google, but over the past 10 years has been adopted by quite a large community. The really simple version is it lets you run containers across a cluster, while the configuration of that cluster is done by manipulating declarative resources.

What's the Plan

I have started to put together a basic design. Over the next few days (weeks?), I'd like to start to assemble some of the core components.

I've also done the most important part, I've figured out a scheme for naming all of the components. Kubernetes is the Greek word for Helmsman or navigator. As a result, a lot of the things in the Kubernetes ecosystem are ship-themed, or otherwise related to ocean navigation. Being more of a train nerd than a ship one (even though I live on a houseboat, that theme is already taken), I've settled on a number of train-based names for the various sub-systems that this project will be made of.

I am, therefore, building the Conductor container platform. It will be comprised of a Controller communicating with a Railway of Engines. Each Engine represents a physical or virtual server in the Railway. Each Engine pulls a consist of Wagons, which can carry one or more Containers each.

Hopefully this little cheat sheet clears up any confusion. If not, perhaps seeing the system come together over the next few posts will help.

Kubernetes    => Conductor
Control Plane => Controller
Node          => Engine
Pod           => Wagon
Cluster       => Railway
Scheduler     => Dispatcher
kubelet       => engined
kubectl       => trainctl

Design and Constraints

If I tried to cover the entire scope of what Kubernetes can do, it'd take me forever. Kubernetes has been around for a very long time, nearly 10 years at the time of writing. I need to target a small but representitive subset of functionality. Specifically, I'd like Conductor to have:

A separate control plane from the nodes it controls
The ability to scale the cluster at runtime
An equivelant to Kubernetes Pods, ReplicaSets, Deployments, and maybe DaemonSets
Some equivelant to Kuberetes ConfigMaps (and perhaps Secrets)
Some ability to define custom resources and run custom operators and controllers

I plan on writing the entire system in Rust. I've been using it for all of my personal projects lately, and have been enjoying it. It'll live on Github, like all of my other public projects, at https://github.com/ucosty/conductor.

Conductor Phase 1

This block diagram rougly represents what I'd like to get out of phase 1 of this project. Some kind of working command-line client (trainctl) talking to a single Controller instance. The Controller instance uses Redis as its persistence store. It can have one or more Engine nodes registered. On each Engine, a number of Wagons can be scheduled. Each Wagon can run a container.

Theory of Operation

The core of Conductor is ceneterd around a control loop. As a user, you get the cluster to run services by applying changes to the desired configuration. This configuration change is picked up by controllers, which compare the desired state to the actual running state of the cluster. The controllers then apply the appropriate changes, such as creating or deleting resources. The feedback loop is closed by the controller when the cluster's actual state reflects the desired state.

Conductor Control Loop

It is also possible for the controller to instance new desired configuration, which can be reconciled through a different controller. As an example, a user could create a ReplicaSet. A ReplicSet creates n replicas of a given Wagon. This is done by having the ReplicaSet update the desired state configuration to reflect the required number of wagons configured in the system. The existence of these new Wagons will then be reconciled by the Wagon Controller. The Wagon Controller will, in turn, ensure the right container configurations are created and that the Wagons are assigned to Engines.

Building the System

Now I have a rough plan and an idea of how the system should work, it's time to start working on building the thing. My plan is to write a chunk of code, get a section of the system working, upload it to GitHub, and then write about it. This means the blog will lag behind the code repository by a little bit, but the blog will always catch up.

ucosty.io