Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libirmin: reuse a single eio scheduler across calls #2318

Open
wants to merge 3 commits into
base: eio
Choose a base branch
from

Conversation

art-w
Copy link
Contributor

@art-w art-w commented May 17, 2024

(builds on top of the other Eio PRs, only the last commit 121309c is new)

The port to Eio of libirmin inherited a defect from the previous integration with the Lwt scheduler: Every calls from C into OCaml land would start a fresh scheduler, run the Irmin function, stop the scheduler and return the result to the C world. Not only is it inefficient to start/stop the scheduler so often, stopping Eio will also close any opened database files thanks to its strict resource management... which was not an issue before as Lwt allowed us to leak the open files across calls!

To address this, @talex5 recommended that we spawn a new domain to run the Eio scheduler on, with the C thread submitting request to it and blocking until the response comes back... but since libirmin doesn't expose any "async" operations to C, we would always have a single domain active at a time (either the C thread or the Eio one).
So it was slightly easier and more fun to flex some effect handlers to achieve the same outcome without a second domain:

  • We start the Eio scheduler once on the first C call to an Irmin function, but we don't let Eio stop as we suspend it by performing an effect and save its continuation for later
  • The next C call to Irmin will resume the suspended Eio scheduler, just long enough to run the Irmin function in its scope, then suspend the scheduler again, etc

cc @kayceesrk "Eio dawg, I heard you like schedulers"

@kayceesrk
Copy link
Contributor

kayceesrk commented May 21, 2024

Nice use of effect handlers 👍.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants