Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preload batch size is artificially reduced as we count in-progress rows with a pending AS action twice #6473

Open
MathieuLamiot opened this issue Mar 11, 2024 · 2 comments

Comments

@MathieuLamiot
Copy link
Contributor

Context
I identified the issue when rewriting integration tests for #6396
When deciding on the preload batch size, two limitations are implemented:

  • We reduce the max. batch size by the number of actions in the AS queue in process_pending_jobs()
  • We reduce the batch size by the number of in-progress actions in the DB in get_pending_jobs()
    As a result:
  • If a page has a pending AS action, and is in-progress in the DB (which is an expected behavior), then it is counted twice as a reduction of the batch size.

Expected behavior
The effective maximum batch size must be rocket_preload_cache_pending_jobs_cron_rows_count - number of pending actions in the AS queue - in_progress rows that don't have a pending actions in the AS queue.

Acceptance Criteria
Test case InProgressShouldNotExceedMaxQueue must be re-written exactly as follows and passing:

  • rocket_preload_cache_pending_jobs_cron_rows_count must be set to 4
@piotrbak
Copy link
Contributor

@MathieuLamiot This is only about the tests, right?

@MathieuLamiot
Copy link
Contributor Author

MathieuLamiot commented Jun 28, 2024

No it's actually impacting the way we do preload. It is still valid and will be even after #6396. The impact is that preload can be slower than expected as we count twice an in-progress preload in the batch size.
So if you start a new batch (expected: 45 items) but there are already 6 in-progress, I think we only add 33 new items instead of 39.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants