riscv: Initial support with v6.5 bindings

[endeneer]

Generated based on Linux tag v6.5.

Summary of the PR

Related: #80

The journey of a thousand miles begins with a single step.

The first step of porting Rust virtualization stack to RISC-V is to enable kvm-bindings :-)
The next step will be porting kvm-ioctls.

Currently there is no CI for RISC-V, I will continue studying the rust-vmm-ci docs tomorrow.

Requirements

Before submitting your PR, please make sure you addressed the following
requirements:

• All commits in this PR are signed (with git commit -s), and the commit
  message has max 60 characters for the summary and max 75 characters for each
  description line.
• All added/changed functionality has a corresponding unit/integration
  test.
• All added/changed public-facing functionality has entries in the "Upcoming
  Release" section of CHANGELOG.md (if no such section exists, please create one).
• Any newly added unsafe code is properly documented.

[roypat]

Hi @endeneer, welcome to rust-vmm! Thank you for your contribution :) We've been getting questions about Risc-V support every now and again, so its nice to see something happening in that direction. I was just wondering, do you have an outline of all the changes that will be required across rust-vmm crates to support Risc-V? E.g. CI/testing, how to integrate Risc-V support into other fundamental crates such as https://github.com/rust-vmm/linux-loader or https://github.com/rust-vmm/vm-memory. Just so we have an overview over where these efforts will be going :)

Also, it'd be nice to have all kernel bindings be generated from the same kernel version. Currently, x86_64 and aarch64 are generated from 6.2, can be use the same base for Risc-V?

[rbradford]

Hi @endeneer, welcome to rust-vmm! Thank you for your contribution :) We've been getting questions about Risc-V support every now and again, so its nice to see something happening in that direction. I was just wondering, do you have an outline of all the changes that will be required across rust-vmm crates to support Risc-V? E.g. CI/testing, how to integrate Risc-V support into other fundamental crates such as https://github.com/rust-vmm/linux-loader or https://github.com/rust-vmm/vm-memory. Just so we have an overview over where these efforts will be going :)

Initially I think cross-compiling the bindings would be a good start: a cross compiling GH action example can be found here - https://github.com/rust-vmm/acpi_tables/blob/main/.github/workflows/quality.yaml#L14

vm-memory shouldn't need any changes, for linux-loader the PE loader from aarch64 can be reused for RISC-V

Also, it'd be nice to have all kernel bindings be generated from the same kernel version. Currently, x86_64 and aarch64 are generated from 6.2, can be use the same base for Risc-V?

I agree; I would like to see the same version used consistently.

[rbradford]

Sorry for being ambiguous - you need to bump all the architectures to 6.5. Kernel headers from 6.2 are not really sufficient for virtualisation on RISC-V (no AIA/IMSIC.)

[endeneer]

Sorry for being ambiguous - you need to bump all the architectures to 6.5. Kernel headers from 6.2 are not really sufficient for virtualisation on RISC-V (no AIA/IMSIC.)

It would be nice to have AIA/IMSIC/IOMMU RISC-V support, but IMHO v6.2 should be more than enough for kvm-ioctls porting (in comparison to v5.19 kernel which I used while working on kvmtool).

[rbradford]

Sorry for being ambiguous - you need to bump all the architectures to 6.5. Kernel headers from 6.2 are not really sufficient for virtualisation on RISC-V (no AIA/IMSIC.)

It would be nice to have AIA/IMSIC/IOMMU RISC-V support, but IMHO v6.2 should be more than enough for kvm-ioctls porting (in comparison to v5.19 kernel which I used while working on kvmtool).

It's really not useful since without IMSIC you can't do MSI which really limits the usefulness for PCI. It would be fine for something like Firecracker but Cloud Hypervisor relies on PCI for the virtio devices. If your look at the other Rust based VMM that supports RV64 (CrosVM) it's bindings are generated from a kernel branch including the AIA changes.

By not using the latest version you're just creating more work in the future when we have to update to v6.5 to make the bindings useful.

[roypat]

Why do we not do the FamStruct stuff for risc? :o If they're simply not needed, then probably this comment should also be removed

[endeneer]

Not seeing any FAM usage in RISC-V KVM API in Linux v6.5, so I have removed the comment altogether.

[roypat]

I don't think this is necessary anymore, the linked issue got resolved a while ago, and the bindgen version you used to generate the bindings does not generate unsound derefs anymore (I think with a recent compiler version they actually became a hard-error, so we must've fixed it for the other bindings at some point, too)

[endeneer]

Ah I see, I have removed it.

[arigo-vos]

Generated based on Linux tag v6.5.

Summary of the PR

Related: #80

The journey of a thousand miles begins with a single step.

The first step of porting Rust virtualization stack to RISC-V is to enable kvm-bindings :-) The next step will be porting kvm-ioctls.

Currently there is no CI for RISC-V, I will continue studying the rust-vmm-ci docs tomorrow.

Requirements

Before submitting your PR, please make sure you addressed the following requirements:

• All commits in this PR are signed (with git commit -s), and the commit
  message has max 60 characters for the summary and max 75 characters for each
  description line.
• All added/changed functionality has a corresponding unit/integration
  test.
• All added/changed public-facing functionality has entries in the "Upcoming
  Release" section of CHANGELOG.md (if no such section exists, please create one).
• Any newly added unsafe code is properly documented.

Dear @endeneer,

Great that you have done this first step in adding support for RISC-V in rust-vmm. In the context of the Vitamin-V EU project we are also working on this. For the time being we are updating vmm-reference to the latest crates; then the plan is to extend it and add the necessary bits to run a RISC-V Linux guest. For instance, I believe we must first add PLIC emulation (or AIA support...) but we'd like to discuss this in the next weekly calls. So, if you plan to add RISC-V bindings and IOCTLs it will be great! Do you plan to make other contributions? Thanks!

(Sorry for abusing of your PR's message thread, but I did not know where else to contact you :) maybe Slack is a better place?)

[endeneer]

@arigo-vos
Hi, happy to know that I'm not alone in this journey!

Here is the summary of what I've done so far for RISC-V:

1. kvm-bindings

• PR is available, changes requested, I will rework it this week.

2. kvm-ioctls

• Cross-compilation is successful, but take note that one more PR needs to be merged.
• No riscv-specific ioctls has been added yet.

3. linux-loader

• PR is available, changes requested, I will rework it this week.

4. rust-vmm-container

• This is for CI purpose. Unlike x64/ARM, there is no RISC-V server for CI use. qemu-user-static can't be used since /dev/kvm of the same architecture is required. Therefore, I proposed using running QEMU with custom Linux kernel compiled with CONFIG_KVM=y.
• PR is available, but free Git workflow runner might encounter space limit when building the container. I will look into this potential issue in the next month.

For instance, I believe we must first add PLIC emulation (or AIA support...) but we'd like to discuss this in the next weekly calls.

I have not started working on vmm-reference so far. From the first glance, yes, we need irqchip since Linux 8250 serial driver expects interrupts in devicetree binding. I suggest we should add AIA in-kernel irqchip in kvm-ioctls, this is to avoid writing additional code to emulate PLIC/OpenSBI console in vmm-reference.

Do you plan to make other contributions? Thanks!

My original aim is to enable Kata Containers on RISC-V but diverted due to rust-vmm being one of the dependencies. Once the PR to the above 4 repos have been merged, I plan to work on Kata Containers again, and there ought to be more improvement to rust-vmm RISC-V support along the way.

(Sorry for abusing of your PR's message thread, but I did not know where else to contact you :) maybe Slack is a better place?)

I just joined Slack (@Tan En De), I'm open for collaboration, feel free to message/call me for further discussion. I have knowledge in writing KVM-based VMM from scratch and have experience contributing to kvmtool. 🐧

[arigo-vos]

@arigo-vos Hi, happy to know that I'm not alone in this journey!

Here is the summary of what I've done so far for RISC-V:

1. kvm-bindings

• PR is available, changes requested, I will rework it this week.

2. kvm-ioctls

• Cross-compilation is successful, but take note that one more PR needs to be merged.
• No riscv-specific ioctls has been added yet.

3. linux-loader

• PR is available, changes requested, I will rework it this week.

4. rust-vmm-container

• This is for CI purpose. Unlike x64/ARM, there is no RISC-V server for CI use. qemu-user-static can't be used since /dev/kvm of the same architecture is required. Therefore, I proposed using running QEMU with custom Linux kernel compiled with CONFIG_KVM=y.
• PR is available, but free Git workflow runner might encounter space limit when building the container. I will look into this potential issue in the next month.

For instance, I believe we must first add PLIC emulation (or AIA support...) but we'd like to discuss this in the next weekly calls.

I have not started working on vmm-reference so far. From the first glance, yes, we need irqchip since Linux 8250 serial driver expects interrupts in devicetree binding. I suggest we should add AIA in-kernel irqchip in kvm-ioctls, this is to avoid writing additional code to emulate PLIC/OpenSBI console in vmm-reference.

Do you plan to make other contributions? Thanks!

My original aim is to enable Kata Containers on RISC-V but diverted due to rust-vmm being one of the dependencies. Once the PR to the above 4 repos have been merged, I plan to work on Kata Containers again, and there ought to be more improvement to rust-vmm RISC-V support along the way.

(Sorry for abusing of your PR's message thread, but I did not know where else to contact you :) maybe Slack is a better place?)

I just joined Slack (@Tan En De), I'm open for collaboration, feel free to message/call me for further discussion. I have knowledge in writing KVM-based VMM from scratch and have experience contributing to kvmtool. 🐧

Thanks for the prompt and detailed response. I agree about AIA support, however we'd like to test it as soon as possible on real hardware (at the moment, as far as I know, nothing supports the H extension). So I'm wondering whether the new hardware that will come will support both H and AIA or only H. Depending on this, we might also consider to add PLIC emulation. Anyway, let's continue the discussion on Slack to not clobber this thread.

[danielhb]

Hi @endeneer @arigo-vos ! I crossed this MR after starting an attempt to port kvm-ioctl and kvm-bindings to RISC-V myself. In my case I'm using a full emulated QEMU RISC-V host running Ubuntu 22.

What's the status here? Anything I can do to help? I see the discussions about PLIC and AIA but I understand this isn't something that an initial port should worry about.

Thanks!

[arigo-vos]

Hi @endeneer @arigo-vos ! I crossed this MR after starting an attempt to port kvm-ioctl and kvm-bindings to RISC-V myself. In my case I'm using a full emulated QEMU RISC-V host running Ubuntu 22.

What's the status here? Anything I can do to help? I see the discussions about PLIC and AIA but I understand this isn't something that an initial port should worry about.

Thanks!

Hello @danielhb ,

We have been working on the kvm-ioctls crate for a while, and we are pretty confident we have most of the pieces to run Linux. We are now wiring all up in vmm-reference to eventually attempt to instantiate a Linux guest! If all goes well, next week we may post about it. Then, we will publish the work done on kvm-ioctls. Stay tuned!

[rbradford]

Hi @endeneer @arigo-vos ! I crossed this MR after starting an attempt to port kvm-ioctl and kvm-bindings to RISC-V myself. In my case I'm using a full emulated QEMU RISC-V host running Ubuntu 22.

What's the status here? Anything I can do to help? I see the discussions about PLIC and AIA but I understand this isn't something that an initial port should worry about.

Well my understanding is that without AIA we can't do MSI and from my perspective that also limits our ability to do virtio-pci as Cloud Hypervisor only does virtio-pci with MSI(-X). Anyway - the required functionality is now merged into Linus's tree so this should be in v6.9 and from there we can update this PR using that version.

[roypat]

Hi all,
We have merged riscv64 bindings generated from kernel version 6.9 in #106. I am going to go ahead and close this PR as resolved by #106. Thanks again to everyone for your contributions here!