Down a Rabbit Hole: The Importance of CI/CD

Down a Rabbit Hole: The Importance of CI/CD

Open source developers working with Arm have to go down a lot of rabbit holes, including the intricate (and often unforeseen) dependencies and interdependencies that can pop up during a build.

With build systems, the thing you do first is in almost every system is compute dependency analysis. Then once you have some understanding of all there is to do, you decide what tasks to perform in parallel — especially on multi-core systems. 

The problem is, what to do first in a complex environment is not always obvious. What you need for success is an evolved approach — one that doesn’t depend on proof of concept (POC), but that focuses on repeatable success and scalability.

Once things work, if you want them to rinse and repeat, then you enter into a world of CI/CD. Here, not only does the project have to work during POC, but it also has to continue to work over time for your architecture. 

Having collaborated on more than 100 projects, some of the CI/CD challenges specific to Arm that I’ve run into include lack of native support, imperfect emulation in QEMU, complexity of cluster build environments, and of course, a poor understanding of dependencies up and down the chain. 

Did Someone Say Rabbit?

Pulling a rabbit out of your hat can get tricky to say the least.

When I think of rabbits, I think of creatures that are fast and quick to multiply. Fast-growing active projects are similar in nature, and when combined with a finite amount of resources can lead to endless tunneling in deep, dark rabbit holes. 

These can be overcome with much diligence and effort (so much digging!) — plus a few juicy carrots (read: tools that help make the journey more enjoyable) to enable fast-path growth and help make solving all those build problems much easier. 

Here are five of my favorite carrots for accelerating tricky projects:

  • Use modern tooling that supports multi-architecture builds directly. For example, Docker’s buildx lets you build multi-arch images, link them together with a manifest file, and push them all to a registry using a single command. Artur Klauser has a nice tutorial on building multi-architecture Docker images with buildx that he published to Medium earlier this year.


  • Keep your Dockerfiles under control with hadolint, a smart lint tool that helps you keep your code within standard best practices. Hadolint will warn you when your Dockerfile uses constructions that are ambiguous, risky, or unportable, and keeping it happy will keep you happy in the long run. Bas Harenslack wrote an introduction to Linting your Dockerfile with Hadolint for Go Data Driven that’s a good place to start.


  • Another lint tool for your build system is shellcheck, an open source static analysis tool that automatically finds bugs in your shell scripts. Debugging shell code can be awkward because of its venerable syntax, and shellcheck remembers for you the various quoting conventions necessary to avoid surprises. Andy Crouch has a useful review of shellcheck, showing some common errors that it catches. 



  • Learn about NixOS, a purely functional operating system that has a build system (nixpkgs) with extensive caching, reproducible results, and fast and exact repeatability. NixOS deals with dependency conflicts by letting you pin every build to a precise description of what it depends on. Graham Christensen’s weblog is an ever-entertaining set of stories about NixOS, including his recent Erase Your Darlings about keeping your work environment in order by erasing it and recreating it (quickly!) whenever you get started.

There’s a lot to learn about build systems, and dozens of examples of good practices and innovative tools that I didn’t include here. If you have a favorite, please share it with me ([email protected]) to include in a future feature or on social media!

You can unsubscribe at any time.

Explore Topics