Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial riscv64 support for rust-vmm-container #91

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

endeneer
Copy link

This pull request is to introduce riscv64 Docker image for rust-vmm CI purpose.

Unlike x86/ARM CI test, we don't have native server for RISC-V.
As a workaround, QEMU can be used so that we can run RISC-V CI on x86/ARM CI server.

The CI flow using QEMU looks like this:

  1. Pass the repo to be tested as a volume to Docker container.
  2. In the container, cross-compile the repo (I did try Rust RISC-V native toolchain, but it seems unstable at the moment).
  3. The Docker container proceed to launch a QEMU RISC-V guest and pass the cross-compiled repo to the guest using 9pfs.
  4. The guest boots, and during init, a custom systemd service (test.service) will mount the 9pfs, which then searches and runs all the cross-compiled test binaries in that 9pfs mount point.
  5. The guest writes to virt test MMIO device using devmem utility to trigger QEMU exit with custom exit code.
  6. The exit code is passed along the exit path from QEMU to Docker container and back to host CI server.

To better explain the CI flow, I also made a YouTube video:
https://youtu.be/r2OhfmPq74U

  • All commits in this PR are signed (with git commit -s), and the commit
    message has max 60 characters for the summary and max 75 characters for each
    description line.
  • All added/changed functionality has a corresponding unit/integration
    test.
  • All added/changed public-facing functionality has entries in the "Upcoming
    Release" section of CHANGELOG.md (if no such section exists, please create one).
  • Any newly added unsafe code is properly documented.

@andreeaflorescu
Copy link
Member

@endeneer thanks for adding this support in rust-vmm. I think this is pretty cool!

We had some problems with the space on the git runners because of running in separate RUN commands. That's why we added all required tools in a single RUN command. See 5108073. It might be that you'll need to update the Docker file to have it similarly as in the other ones to not run in the space problem.

I also notice that the CI is not running for the newly added container. I guess this is because we need to update the .github/workflows as well to include the scripts related to building the RISC-V container. Can you update it and that way we'll also see if the build is successful?

@endeneer
Copy link
Author

endeneer commented Dec 19, 2023

Thanks @andreeaflorescu, I have updated the Dockerfile as suggested.
I have successfully tested build and push in my own repository:

The Buildkite CI pipeline generation is ready too (rust-vmm/rust-vmm-ci#148).

PS
🎅 🎄 Merry Christmas everyone, looking forward to adding more riscv64 support for rust-vmm next year, but for now, just have fun! ☃️

README.md Outdated Show resolved Hide resolved
@TexasOct
Copy link

TexasOct commented May 15, 2024

I noticed that this pull request hasn't changed in a long time, and I'd like to do some work for it, what can I do for it?

If needed, I will do this work in a new pull request to taking it forward.

There is no RISC-V server at the moment,
so we don't have a native machine to try out RISC-V KVM API.
Therefore, we have to use QEMU to emulate RISC-V machine during unittests.
To support such emulation, Dockerfile.riscv64 builds Docker image that contains
QEMU along with OpenSBI, Linux kernel, and rootfs needed by the guest.

Signed-off-by: Tan En De <[email protected]>
- qemu.sh is for launching QEMU guest from inside Docker container.
- interfaces and resolv.conf are for setting up networking in QEMU guest.
- test.service and test-service.sh is to create systemd service that runs
test during system init.

Signed-off-by: Tan En De <[email protected]>
For riscv64 build/publish/manifest, introduce `ARCH=riscv64` environment variable,
to be used like this: `ARCH=riscv64 ./docker.sh <build/publish/manifest>`.

Signed-off-by: Tan En De <[email protected]>
Add example scripts to show the usage of the riscv64 Docker image:
- docker-build.sh: Build riscv64 rust-vmm-container Docker image.
- docker-run.sh: Enter the container image and see how's the build went.
- docker-test.sh: Run CI for build-gnu-riscv64 and unittests-gnu-riscv64.

Signed-off-by: Tan En De <[email protected]>
Update README.md to inform about the addition of riscv64 rust-vmm-container support.

Signed-off-by: Tan En De <[email protected]>
To handle push/pull request concerning riscv64-related files.

Signed-off-by: Tan En De <[email protected]>
@endeneer
Copy link
Author

Hi @TexasOct , I just updated this PR by resolving the suggestions from maintainer.
At the moment, before rust-vmm starts to accept riscv64-related commits, it's unsure whether we shall wait until Buildkite supports RISC-V agent platform, or use the workaround I provided in this PR (run CI in QEMU on x86 Buildkite agent platform). Maybe we shouldn't rush, at the same time keeping an eye on industry news regarding availability of RISC-V servers for CI/CD services.

TimePrinciple added a commit to TimePrinciple/rust-vmm-container that referenced this pull request Aug 5, 2024
A developed version of rust-vmm#91, referenced the idea of environment
preparation for qemu-system-riscv64. Add build scripts for v6.10 riscv64
kernel, qemu-system-riscv64 and opensbi required to boot qemu-system.

With this approach, we are able to run tests inside qemu-system, while
preserving the original output as much as possbile with ssh.

Signed-off-by: Ruoqing He <[email protected]>
@TimePrinciple TimePrinciple mentioned this pull request Aug 5, 2024
4 tasks
TimePrinciple added a commit to TimePrinciple/rust-vmm-container that referenced this pull request Aug 6, 2024
A developed version of rust-vmm#91, referenced the idea of environment
preparation for qemu-system-riscv64. Add build scripts for v6.10 riscv64
kernel, qemu-system-riscv64 and opensbi required to boot qemu-system.

With this approach, we are able to run tests inside qemu-system, while
preserving the original output as much as possbile with ssh.

Signed-off-by: Ruoqing He <[email protected]>
TimePrinciple added a commit to TimePrinciple/rust-vmm-container that referenced this pull request Aug 7, 2024
Add build scripts for v6.10 riscv64 kernel, qemu-system-riscv64 and
opensbi required to boot qemu-system inside docker container.

With this approach, we are able to run tests inside qemu-system, while
preserving the original output as much as possbile with ssh.

This work was inspired by the work done by @endeneer in PR rust-vmm#91, and is
the third draft proceeds rust-vmm#101, rust-vmm#104. It is expected to be replaced by

Signed-off-by: Ruoqing He <[email protected]>
TimePrinciple added a commit to TimePrinciple/rust-vmm-container that referenced this pull request Aug 7, 2024
Add build scripts for v6.10 riscv64 kernel, qemu-system-riscv64 and
opensbi required to boot qemu-system inside docker container.

With this approach, we are able to run tests inside qemu-system, while
preserving the original output as much as possbile with ssh.

This work was inspired by the work done by @endeneer in PR rust-vmm#91, and is
the third draft proceeds rust-vmm#101, rust-vmm#104. It is expected to be replaced by

Signed-off-by: Ruoqing He <[email protected]>
TimePrinciple added a commit to TimePrinciple/rust-vmm-container that referenced this pull request Aug 7, 2024
Add build scripts for v6.10 riscv64 kernel, qemu-system-riscv64 and
opensbi required to boot qemu-system inside docker container.

With this approach, we are able to run tests inside qemu-system, while
preserving the original output as much as possbile with ssh.

This work was inspired by the work done by @endeneer in PR rust-vmm#91, and is
the third draft proceeds rust-vmm#101, rust-vmm#104. It is expected to be replaced by
the second draft rust-vmm#104 in the future which standardise `riscv64`.

Signed-off-by: Ruoqing He <[email protected]>
TimePrinciple added a commit to TimePrinciple/rust-vmm-container that referenced this pull request Aug 8, 2024
Add build scripts for v6.10 riscv64 kernel, qemu-system-riscv64 and
opensbi required to boot qemu-system inside docker container.

With this approach, we are able to run tests inside qemu-system, while
preserving the original output as much as possbile with ssh.

This work was inspired by the work done by @endeneer in PR rust-vmm#91, and is
the third draft proceeds rust-vmm#101, rust-vmm#104. It is expected to be replaced by
the second draft rust-vmm#104 in the future which standardise `riscv64`.

Signed-off-by: Ruoqing He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants