Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert memtable rotation to a channel message #78

Open
marvin-j97 opened this issue Sep 21, 2024 · 0 comments
Open

Convert memtable rotation to a channel message #78

marvin-j97 opened this issue Sep 21, 2024 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed performance

Comments

@marvin-j97
Copy link
Collaborator

marvin-j97 commented Sep 21, 2024

Currently the memtable rotation happens in the write hot path when the memtable size threshold is reached https://github.com/fjall-rs/fjall/blob/main/src/partition/mod.rs#L748-L749.
The performance hit mostly comes from the journal rotation which needs to fsync to disk.
It would be better for performance to send a message via channel, announcing that the partition needs to be flushed, nullifying rotation costs inside the write hot path, which should give quite a performance boost for write heavy applications (currently it is slightly slower than Rocks).
The messages should be picked up by a worker that rotates the partition asynchronously, essentially performing the Partition::rotate_memtable (https://github.com/fjall-rs/fjall/blob/main/src/partition/mod.rs#L633) function.

@marvin-j97 marvin-j97 added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed performance labels Sep 21, 2024
@marvin-j97 marvin-j97 pinned this issue Sep 21, 2024
@marvin-j97 marvin-j97 transferred this issue from fjall-rs/lsm-tree Sep 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed performance
Projects
None yet
Development

No branches or pull requests

1 participant