Building Multi Arch Containers With Buildah in GitLab CI

I’m running my own k3s cluster to have some fun with over-engineered solutions. To make it even more interesting, it’s a multi-arch cluster. It contains both x86_64 and aarch64 nodes. Does it sound scary to you? It actually works quite nicely, you just need to make sure that all your container images are also multi-arch.

Let’s look at how these containers can be built using buildah and GitLab CI.

Building single-arch containers with Buildah #

I generally prefer to use podman nowadays since it’s always up-to-date in Fedora and CentOS Stream. Due to this reason, its sibling, buildah, was a natural choice for this task. Buildah ships its own container that contains everything you need to build images. It must be run in a privileged container but fortunately, gitlab.com gives you one.

All you need to do to build a container on every push to a main branch is to use the following pipeline:

build:
  stage: build
  image:
    name: quay.io/buildah/stable
  script:
    - buildah login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" ${CI_REGISTRY}
    - buildah build --tag ${CI_PROJECT_NAME} .
    - buildah push ${CI_PROJECT_NAME} docker://${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}
  only:
    - main

Simple, right? You don’t even have set to any variables, everything is set up for you by default. The first command logs in GitLab Container Registry. The second one builds the container from Containerfile located in the project’s root directory. The final command pushes the built image into GitLab Container Registry and tags it with the commit’s SHA.

Going multi-arch #

The previous paragraph achieved building just one container image for one architecture. Let’s go multi-arch as I promised!

Firstly, it’s important to note that when going multi-arch, you have to ensure that your base images are already built for all architectures that you need, otherwise, this obviously won’t work. Fortunately, that is slowly becoming the new standard - let’s take Golang containers for example: Both the ones based on UBI and the ones hosted on Docker Hub are already being built for plenty of architectures.

Now it’s needed to explain how buildah is going to run any binaries that are inside the container and thus might be built for a different architecture: To enable this, we will need to install the QEMU User space emulator. It allows you to (mostly) seamlessly run binaries compiled for a different CPU architecture than the one that runs your operating system. Note that this is just an emulation so it’s surely much slower than native build. Nevertheless, for building small containers, it’s actually not a big deal.

This is the modified pipeline for building both x86_64 and aarch64 containers:

build:
  stage: build
  image:
    name: quay.io/buildah/stable
  script:
    - dnf install -y qemu-user-static
    - buildah login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" ${CI_REGISTRY}
    - buildah build --jobs=4 --platform=linux/amd64,linux/arm64 --manifest ${CI_PROJECT_NAME} .
    - buildah manifest push --all --format v2s2 ${CI_PROJECT_NAME} docker://${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}
  only:
    - main

It doesn’t look much more complicated, right?

The first line of the script just installs qemu-user-static to enable buildah to use the QEMU emulator.

The build command adds three things: --manifest instructs buildah to add all built images to an image manifest so they can all have the same tag. --platform controls for which platforms will the image be built. Note that this is using the architecture names as defined in Golang which might be different from what you are used to. Finally, I use --jobs=4 to parallelize the build. In most cases --jobs=2 should be enough for 2 architectures but in this particular case I was running a multi-stage build so by using --jobs=4 I allowed buildah to build everything in parallel if possible.

The push command is also different: Instead of the plain old push, I use manifest push which signifies that I want to push the whole manifest instead of just one image. The important argument is --format v2s2. If it’s not used, an OCI format will be used instead which is unfortunately not yet implemented by GitLab.

And that’s it! You can inspect a full project using this method here: https://gitlab.com/ondrejbudai/hello-kubernetes

If multi-arch isn’t possible #

In certain cases, you might not be able to build multi-arch containers. For example, your base doesn’t ship the needed architecture, or the build is just too slow to be practical. In this case, you can use node selectors to restrict the pod scheduling to just the nodes of the preferred architecture. Fortunately, Kubernetes nowadays automatically label nodes with their architecture so nothing else is needed than adding a simple nodeSelector to your deployments:

nodeSelector:
  kubernetes.io/arch: amd64