27 May 2021
Earlier this year I built a web-based garage door controller using Rust for the Raspberry Pi called Normally Closed. My deployment includes a Pi 3b and a Pi Zero which are ARMv7 and ARMv6 devices, respectively. I deploy services with Docker and wanted to continue using it here for simplicity.
Getting all of this set up and working together was not easy. There’s also a lot of words in that title which might not mean much to you. That’s okay! This is the blog post that I needed two weeks ago, so let’s take a look at the steps required to accomplish this task.
Rust has excellent facilities for cross-compiling and static linking through Cargo. I got started following this guide on cross-compiling Rust for the Raspberry Pi.
The guide recommends using the
armv7-unknown-linux-gnueabihf Rust target which would support my Pi 3b. For the Pi Zero we can infer from Rust’s platform support list that we need
arm-unknown-linux-gnueabihf. However, these targets dynamically link against GNU libc whereas I wanted to statically link with musl. Referencing the platform support list again we can find the
arm-unknown-linux-musleabihf targets to use instead.
$ rustup target add armv7-unknown-linux-musleabihf $ rustup target add arm-unknown-linux-musleabihf
In addition to the target, the linker needs to be changed since it otherwise will use the one from your machine and its architecture. This can be specified in
[target.armv7-unknown-linux-musleabihf] linker = "arm-linux-gnueabihf-ld" [target.arm-unknown-linux-musleabihf] linker = "arm-linux-gnueabihf-ld"
On Ubuntu you can install this linker by running
sudo apt install gcc-arm-linux-gnueabihf. On my Mac I was able to install it with
brew install arm-linux-gnueabihf-binutils. (Despite using the musl compilation target, the GNU linker will still work.)
This is enough to compile working binaries!
$ cargo build --target armv7-unknown-linux-musleabihf --release $ cargo build --target arm-unknown-linux-musleabihf --release
I really like to make my binaries and Docker containers as small as possible. Once again Rust’s Cargo gives us a simple mechanism to achieve this in our
[profile.release] opt-level = 'z' lto = true codegen-units = 1
z instructs the compiler to favor a smaller binary size over performance. This will not be appropriate for anything CPU-intensive, but for a web server which will only see a few interactions per year we don’t require maximum performance.
LTO is short for “link-time optimization” which performs optimization on the whole program rather than locally on individual functions. By virtue of analyzing the whole program it also improves the ability to remove dead code.
codegen-units setting reduces the parallelism of Cargo to allow compilation to occur in a single unit and be optimized as a single unit. This allows compilation and optimization to have the maximum impact by always seeing the entire program.
Having starting with only the Pi 3b and needing ARMv7, getting the build going in Docker was not too difficult. We basically just run the commands from above to produce the binary and then copy that into an Alpine container.
FROM rust:1.52.1 AS rust RUN rustup target add armv7-unknown-linux-musleabihf RUN apt-get update && apt-get -y install binutils-arm-linux-gnueabihf WORKDIR /app COPY .cargo ./.cargo COPY Cargo.toml Cargo.lock .rustfmt.toml ./ COPY src ./src RUN cargo build --release --target armv7-unknown-linux-musleabihf FROM --platform linux/arm alpine:3.12 WORKDIR /app COPY --from=rust /app/target/armv7-unknown-linux-musleabihf/release/normally-closed ./ # ENTRYPOINT setup...
This container can be built with the regular
docker build . command.
Adding a second architecture complicates things significantly. Docker does support containers which are built for multiple architectures through Docker buildx.
However, unlike the buildx examples, we cannot naively run
docker buildx build --platform linux/arm/v7,linux/arm/v6 . and have it just work. For one, the Rust container is not available for those architectures. But even if it were, we still need to specify the custom compilation target per architecture due to our desire to use musl.
The first step towards making this work is having the Rust container always use the architecture of the machine on which it is running. This is similar to having run Cargo directly on our machine before, and it works because Rust is already allowing us to cross-compile to ARM.
-FROM rust:1.52.1 AS rust +FROM --platform=$BUILDPLATFORM rust:1.52.1 AS rust RUN rustup target add armv7-unknown-linux-musleabihf ⋮
BUILDPLATFORM argument is documented as one which is available by default in the global scope of a Dockerfile.
With Rust always running on our host architecture we still need to vary the Rust target which is used for cross-compilation. Also present in the list of default arguments in Docker is
TARGETPLATFORM which will contain either
linux/arm/v6 in our case. We can use a
case statement to determine the associated Rust target.
FROM --platform=$BUILDPLATFORM rust:1.52.1 AS rust +ARG TARGETPLATFORM +RUN case "$TARGETPLATFORM" in \ + "linux/arm/v7") echo armv7-unknown-linux-musleabihf > /rust_target.txt ;; \ + "linux/arm/v6") echo arm-unknown-linux-musleabihf > /rust_target.txt ;; \ + *) exit 1 ;; \ +esac RUN rustup target add armv7-unknown-linux-musleabihf ⋮
We write the value to a file since
export does not work and there’s not really another mechanism for passing data between build steps. A read of that file replaces the hard-coded targets in the steps that follow.
⋮ esac -RUN rustup target add armv7-unknown-linux-musleabihf +RUN rustup target add $(cat /rust_target.txt) RUN apt-get update && apt-get -y install binutils-arm-linux-gnueabihf ⋮ COPY src ./src -RUN cargo build --release --target armv7-unknown-linux-musleabihf +RUN cargo build --release --target $(cat /rust_target.txt)
The Alpine build stage also references the target in the source folder of the binary. Rather than worry about passing along the file which holds this value, an easy workaround is to copy the binary to a location which does not contain the target name.
⋮ RUN cargo build --release --target $(cat /rust_target.txt) +# Move the binary to a location free of the target since that is not available in the next stage. +RUN cp target/$(cat /rust_target.txt)/release/normally-closed . ⋮
The Alpine build stage can now remove the target platform and copy from the new location.
⋮ -FROM --platform linux/arm alpine:3.12 +FROM alpine:3.12 WORKDIR /app -COPY --from=rust /app/target/armv7-unknown-linux-musleabihf/release/normally-closed ./ +COPY --from=rust /app/normally-closed ./ # ENTRYPOINT setup...
At this point we have a fully-working, cross-compiling, static-linking, multi-architecture Docker container built from Rust!
$ docker buildx build --platform linux/arm/v7,linux/arm/v6 . [+] Building 147.9s (34/34) FINISHED
You can see the result reflected on the Docker Hub listing as compared to the latest release:
The full and final
Dockerfile can be found here for reference. The repository also contains GitHub Actions setup for building the standalone binaries as well as the multi-architecture Docker container.
Hopefully this helps someone! It was a couple nights of piecing together all the steps for me. And hey if you have a garage door and a spare Pi lying around maybe try out Normally Closed!
— Jake Wharton