Manjaro on the Radxa CM3
Recently I purchased a Turing Pi cluster baseboard. More accurately, I backed the project on Kickstarter the day it launched, as I have wanted to build a low-power cluster to play with kubernetes and other distributed computing systems for some time. Previous efforts hadn’t really gone well (it turns out that the Pi Zero does not make for a good kubernetes node with a mere 512M of RAM) so something with more power was appealing. I wasn’t sure what to use to populate it, as Pi CM4s are virtually unobtainable at the time of writing and some appealing new alternatives are coming to market what feels like daily. A little bit of research and I settled on some Radxa CM3s, for a couple of reasons; they were available and cheap (I paid somewhere around $40 per node if I recall), the community and vendor support for the product seemed reasonably healthy, and they offer a more powerful SoC with more RAM than a Pi CM4.
Now, I’m a big fan of Arch Linux and its derivatives (aside from systemd, but that’s a gripe for another day). Radxa only builds official debian, Android, and Ubuntu images. Manjaro-ARM builds a lot of images for a lot of different devices, but sadly, not (yet) for the Radxa CM3. They have a few other Radxa products in their build pipeline though, so I was hoping I could undertake the work to get it ported. The first step, of course, was to figure out what needed to be done.
The Manjaro-ARM Build System
First, a warning: This is not meant to be a comprehensive, accurate, or even up-to-date explainer for the Manjaro ARM build system. For that, I would recommend reading the manjaro-arm-tools README and perhaps taking a look at the github actions for their various image repositories to see how these tools are used.
The Manjaro ARM build system consists of two main scripts, at least for my purposes:
buildarmpkg, a script that uses qemu to build packages for the arm architecture in a managed clean manjaro rootfs - psst, I would highly recommend taking a look at the-koption so you don’t have to re-download the entire rootfs every timebuildarmimg, a script that combines a rootfs and bootloader into a flashable image for a given device
Looking at buildarmimg, it takes a -d flag to pass which device you are building for. These devices are populated based on profiles that are defined in /usr/share/manjaro-arm-tools/profiles/arm-profiles/devices/ (which in turn, is installed from the manjaro-arm-tools package that builds them from their GitLab repository). The profiles themselves are incredibly straightforward: they are simply a list of packages, one per line, that are installed to the rootfs when creating the image for that device. Glancing at a number of profiles, one can see that this generally includes necessary firmware, bootloaders, the linux kernel, and post-install packages.
The only remaining piece that sadly isn’t modular is the image format for a specific device. The buildarmimg script sources a functions.sh that implements most of the logic. There, we find a simple bash case match to determine which commands are used to partition the image, and later on, the same thing to flash the bootloader to that image.
So, in summary: we have learned we need a functional linux kernel and bootloader, possibly device-specific firmware, and knowledge of the structure of the final image so that the SoC can find and start the bootloader, which can in turn load the linux image.
U-Boot for the RK3566
U-Boot is an open-source bootloader targeting embedded platforms. It is able to make use of DeviceTree sources to dynamically build for specific devices. Radxa releases these DTSs, so in theory including them and then building U-Boot should be straightforward. Thankfully, someone else already beat me to this punch, and these sources were upstreamed starting in this commit. Additionally, Radxa actually releases official u-boot images for this device, so I opted to use those instead and hopefully save myself some future head scratching.
I went ahead and threw a quick PKGBUILD together based on some of the other uboot builds in Manjaro-ARM’s GitLab. It’s available in my [personal arch packages repo, though hopefully, I will take the time to get this submitted to Manjaro and upstreamed (which would kind of outmode this whole post, but that’s the price of progress). It’s incredibly simple, simply installing the two necessary U-Boot binaries to the /boot/ directory, and updating the extlinux configuration with default Manjaro kernel parameters. There is also a post-install script that will notify the user that they must manually flash the u-boot binaries to their system whenever the package updates and gives them the commands to do so.
And that’s really all there is to that. Most Manjaro U-Boot packages build from mainline with patches for the device as necessary, which is probably a more secure approach and what should be done here before this gets submitted upstream. For the security-conscious, I don’t believe Radxa has a public audit log for their build artifacts (at least, I couldn’t find any), so using these direct vendor binaries is potentially a nonstarter. Radxa does provide a script to patch, build, and package u-boot yourself, but it targets debian.
Linux for the RK3566 (and specifically, the CM3)
Now, the more interesting (and challenging) piece. I am not acutely familiar with building linux from source for embedded devices and certainly have a lot to learn. The good news is that this device is based on a well-known SoC, and already has linux running on it (via the official debian images). Similar to U-Boot, DTS files are used to provide hardware-specific code necessary for linux to run on a given SoC. These are the same DTS files used to compile U-Boot, so the above is still applicable.
The latest official kernel release that can be downloaded from Radxa, at least, that I could find at the time of this post, is some old patched 4.x kernel. Now, unlike the bootloader, I am much more concerned with running an older kernel, so this was my nonstarter. After some quick poking around, it turns out Radxa provides a script to build both the bootloader and the linux kernel for their devices called bsp. This will automatically pull the latest supported kernel, patch it, build it, and then create a debian package containing the built image.
To make a long story short, I spent several days finding every way to do this incorrectly. I attempted to make an arch linux package to extract the kernel from the .deb, but I never managed to create a bootable image this way. The root of this problem is the fact that I cannot find a way to get a working UART connection with the hardware I have to these CM3s. Even on the stock debian image and toying with every configuration setting I could find, there is absolutely no data from the device on the UART.
As a result, I’m not sure what the error is when U-Boot tries to load the kernel image. As a matter of fact, I wasn’t sure if U-Boot was working at all at this point. At some point during one of my manic midnight sessions, I decided to try pulling the entire boot partition from the official debian image, but keeping the Manjaro rootfs. I knew this wouldn’t work, but I was hoping that perhaps I’d get something to work with. The CM3 has an onboard LED that will flash during kernel startup, and thankfully that flashed for just a brief moment with this configuration; the issue was definitely with loading and starting the kernel, and not with U-Boot or the boot partition!
I tried a number of things here; building based on the patched 4.x kernel, building with the kernel produced by bsp, building the kernel in the same fashion that the official arch kernel is built but applying the patches manually, but nothing seemed to be working. Unfortunately with no output, troubleshooting was near impossible. In hindsight, I think the issue had to do with how I packaged the DTBs and the compiled DeviceTree binaries, but I won’t know unless I can get a UART working on these boards, or some other way to get U-Boot logs.
Fortunately, at some point, I stumbled on a forum post that mentioned that Radxa had submitted their DTSs to upstream linux, and that they had been accepted (which, of course, I can’t find now). I dug a little bit, and sure enough, this commit appears to have added the DTSs, which means that it should be possible to simply use the out-of-the-box Manjaro kernel once it gets merged by simply ensuring the correct DTBs are present at boot! There was only one problem: this was merged in 6.3 RC1, and Manjaro is still shipping 6.2 at the time of writing. There are a number of AUR packages targeting upstream/RC/mainline linux, but merging configs from Manjaro-ARM, Radxa, and the up-to-date kernel proved to be more effort than it was worth (I tried a few times, but never with success).
Finally, I found that Manjaro has an official linux-rc package, and it was indeed updated to 6.3! This package indeed includes the DTBs built from the merged CM3 DTSs, which should be the last piece of the puzzle! So, I updated my device profile once more, built the image (for what felt like the 100th time), and at last…
Success.
Bringing it Together
Ok, I jumped the gun a little with that last bit. We still need to build the image, and there are still several steps to get there. Now that we have U-Boot and Linux for the device, we can run buildarmimg to put together something for the device. This will require authoring a profile for the device, something like the following:
/usr/share/manjaro-arm-tools/profiles/arm-profiles/devices/radxa-cm3:
## Maintained by FoxAmes ##
# Kernel and bootloader stuff
linux-rc
#uboot-radxa-cm3 # Does not exist in the Manjaro-ARM repo, so we can't include it here
plymouth
plymouth-theme-manjaro
# Other device specific packages
crda
linux-firmware
ap6256-firmware
wpa_supplicant
generic-post-install
quartz64-post-install
Additionally, we will need to modify the aforementioned functions.sh so that buildarmimg knows how to create a flashable image with the correct layout for our device. This is pretty simple since the CM3 shares the layout with a number of other devices, so we just need to add our device to the list:
/usr/share/manjaro-arm-tools/lib/functions.sh, line 1097 (approx):
[...]|om1|opi4-lts|radxa-cm3)
dd if=$TMPDIR/boot/idbloader.img of=${LDEV} seek=64 conv=notrunc,fsync > /dev/null 2>&1
dd if=$TMPDIR/boot/u-boot.itb of=${LDEV} seek=16384 conv=notrunc,fsync > /dev/null 2>&1
;;
Now, to the point of the note before, buildarmimg will only pull packages from the Manjaro-ARM repository. For our custom-built U-Boot, we need to include it manually. buildarmimg takes a -i flag to include all packages in a directory. If you simply ran buildarmpkg, that will be /var/cache/manjaro-arm-tools/pkg/aarch64/, but be aware that this will install all packages in the given directory, so make sure it has what you want and only what you want. In my case, I prepared a directory and manually copied the U-Boot package into it, running the build from there.
So, finally, we can build our image:
sudo buildarmimg -d radxa-cm3 -e minimal -b stable -i . -x
As mentioned before, -d specifies our device (it must match the filename of the file you created in profiles), and -i will install packages from a directory (in this case, the current directory). -e specifies the edition (in my case, I don’t want a desktop environment, so minimal is the only choice for me), -b specifies the branch to pull packages from, and -x specifies that we don’t want our final image compressed (so that we can flash it directly). This takes a few minutes, and at the end, you’ll have a flashable image at /var/cache/manjaro-arm-tools/img.
Conclusion
If you’ve followed along this far, thanks! This was an adventure for me and I got a lot more familiar with the DeviceTree system along the way, which is pretty cool. I lucked out in that a lot of work was already done for me- I didn’t have to create the DTSs, the linux image that ultimately worked was simply prebuilt, and the manufacturer builds and releases bootloader images, so the only thing that’s left to do was bring it all together. I half expected an official release from the Manjaro-ARM folks before I got this together and written up, once I find time I will try and adapt the little work here and submit it to upstream so that they can start building automatically.
Download (but also, don’t)
As a final note, and if you’re the type to live dangerously, I’ve uploaded the image I built to a public bucket so you can simply download and flash it yourself. I would highly recommend you consider the full ramifications of doing this- I don’t have a build audit log, and I will not be providing any level of support, guarantee, or warranty for this image. I’m only putting this out there in case it saves heartache and suffering in the time between now and (hopefully) official releases for this device going live.
(sha256: 695ed8eeeb59ca9a220251ea1218f67f62233587cc29e2e290d4379a9d5cd1a5)