Bits of networks

Bits of networks

Networking, system, research

01 May 24

The upstream RISC-V experience: running RISC-V hardware with upstream distros

This post is part of the Upstream RISC-V serie:

A foreword on cfarm.net

In case you don't know about cfarm.net (formerly the GCC Compile Farm), it is the longest-running and most comprehensive compile farm for free software developers. We provide SSH access to machines with many different operating systems and many different CPU architectures. Currently, we have 6 Linux distributions, 5 other OS, and 7 architecture families (with several variants such as big-endian and little-endian ppc64). The goal is to help developers to port, build and debug their projets on less common or hard-to-get hardware.

So, of course, we have been very interested in RISC-V: it's not every day that a whole new CPU architecture gets its place in the landscape, and this specific architecture holds very interesting promises of openness.

The many challenges of a new architecture

Adding RISC-V machines to cfarm.net turned out to be a big challenge: for many years, there was no hardware that was both capable of running Linux and stable enough to provide a compilation service to many concurrent users. Like many projects, we resorted to QEMU emulation. It was working mostly fine (especially because user-mode emulation is reasonably fast on server-class x86 hardware), but it did not provide everything needed: debugging with gdb was a challenge, and microarchitectural details are obviously different in QEMU compared to real hardware.

When capable RISC-V hardware eventually became available, it came with very custom software, because the software ecosystem needed time and hardware to support this new architecture properly. In practice, it meant custom bootloader, custom kernel, and hacked-up Linux distros.

Now, the lowest-levels of software support (libc, compiler, bootloader, kernel) are definitely ready for RISC-V. But support for specific hardware is another story.

Making the case for upstream software and distros

To properly integrate new RISC-V hardware in cfarm.net, we set ourselves a challenge: try to run them with upstream software and distros as much as possible. On our side, this makes the machines easier to manage and is more future-proof. In addition, this offers a more consistent environment to developers using the farm: if their code works on the farm, there is a good chance it will work on different RISC-V hardware and in a different software environment.

This may feel a bit theoretical, but if you have ever worked with vendor SDKs in the embedded world, you know the nightmare it quickly becomes: ancient kernels with very questionable hacks, old compilation toolchains, low-level code that is highly specific to a particular hardware, poor security support, custom and hacky implementations of features that have since been upstreamed in a different way... Unfortunately, the RISC-V ecosystem tends to go a bit into this direction, although it is very far from the worst and we see RISC-V vendors making real efforts to upstream their code.

To give more substance to the argument, here is a list of issues we have encountered so far because of non-upstream code:

Of course, I understand that upstream code does not appear magically, and there is always a fuzzy phase during which code is only available in downstream vendor forks. The point I want to make is the following: we should not be satisfied until the code is available upstream, as it is the only way to ensure its quality and longevity.

Targeted hardware and current plans

Thanks to the awesome support from RISC-V International, we already have received several boards to work on:

  • VisionFive v2
  • Milk-V Pioneer
  • Lichee Pi 4A

We plan to experiment with upstream distros on these boards, document the installation process, submit fixes upstream if required, and generally smooth out the path to get upstream software up and running on this hardware. Then, of course, we will make these boards and upstream software freely available to all cfarm.net developers for the foreseeable future.

Expect more blog articles and news as we progress through this plan.

So far, we mostly target Debian, Alpine Linux and Arch Linux, but we are also exploring other OSes such as OpenBSD.

It should be noted that Ubuntu generally has first-class upstream support for RISC-V boards. Of course they still need to patch the Linux kernel to support the hardware, but this is currently unavoidable. In any case, it's very nice that a distro is able to offer official images with this level of quality. Given this, I believe that we have very little to contribute to Ubuntu to make it better.

Long-term plans

In the long term, we would like to work on vector instructions and virtualisation instructions support, but this is still a long way to go given the lack of hardware supporting these instructions.