Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preload adaptative batch #6427

Merged
merged 54 commits into from
Jul 1, 2024
Merged

Conversation

MathieuLamiot
Copy link
Contributor

Description

When preparing preload batches, the batch size will be adapted based on the average duration of a request to avoid flooding servers with requests if they take too long.
To get the estimate of the request duration, we do a blocking request from time to time when triggering the preload, and measure the timing.

Fixes #6396

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality).

Is the solution different from the one proposed during the grooming?

Yes, on top of the original idea, we added an average mechanism that keeps track of the average request time through transients.

Checklists

Generic development checklist

  • My code follows the style guidelines of this project, with adapted comments and without new warnings.
  • I have added unit and integration tests that prove my fix is effective or that my feature works.
  • The CI passes locally with my changes (including unit tests, integration tests, linter).
  • Any dependent changes have been merged and published in downstream modules.
  • If applicable, I have made corresponding changes to the documentation. Provide a link to the documentation.

Test summary

  • I triggered all changed lines of code at least once without new errors/warnings/notices.
  • I validated all Acceptance Criteria of the related issues. (If applicable, provide proof).
  • I validated all test plan the QA Review asked me to.

Tested locally and on the gamma website by checking transients and scheduled actions.

@MathieuLamiot MathieuLamiot linked an issue Feb 1, 2024 that may be closed by this pull request
@MathieuLamiot MathieuLamiot requested a review from a team February 27, 2024 11:46
inc/Engine/Preload/Controller/PreloadUrl.php Outdated Show resolved Hide resolved
inc/Engine/Preload/Controller/PreloadUrl.php Outdated Show resolved Hide resolved
inc/Engine/Preload/Controller/PreloadUrl.php Outdated Show resolved Hide resolved
@Khadreal Khadreal self-assigned this Feb 29, 2024
@Tabrisrp Tabrisrp requested review from jeawhanlee, Khadreal and a team and removed request for Khadreal and jeawhanlee June 13, 2024 20:48
@Tabrisrp Tabrisrp added type: enhancement Improvements that slightly enhance existing functionality and are fast to implement module: preload labels Jun 13, 2024
/**
* Filter the delay between each preload request.
*
* @param float $delay_between the defined delay.
* @returns float
*/
$delay_between = apply_filters( 'rocket_preload_delay_between_requests', 500000 );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's not been added by this PR, but shouldn't we add a safeguard here ?

@Miraeld Miraeld self-requested a review June 18, 2024 13:37
@Tabrisrp Tabrisrp added this to the 3.16.2 milestone Jun 19, 2024
@MathieuLamiot
Copy link
Contributor Author

MathieuLamiot commented Jun 25, 2024

@wp-media/qa-team To test this issue:

  • Duration of the latest request can be seen in Transients: rocket_preload_check_duration
  • Average duration taken into account is in Transient: rocket_preload_previous_requests_durations
    @piotrbak shared access to some hosts on BitWarden that can provide resource usage plugins built-in:

Cloudways
Hostinger
O2Switch

We can also check CPU usage on rocketlabs (WordOps) for gamma for instance.

It would be great if you could explore a bit how this behaves with an "heavy" website: typically gamma, might want to duplicate it with Duplicator for instance? ; Another approach could be to add some CPU intensive computation in the initialization of WP or of WP Rocket to make as if the website was slow.
It's not easy to reproduce those issues, so it's very exploratory... Apart from non-regressions/errors, the goal would mostly be to see if we're able to see a CPU usage reduction (lower CPU usage, or shorter spikes for instance) on a "heavy" website between trunk and this branch.

Copy link

@hanna-meda hanna-meda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related TP execution Results below:
testrail-report-613.pdf

@hanna-meda hanna-meda added this pull request to the merge queue Jul 1, 2024
Merged via the queue into develop with commit 6367742 Jul 1, 2024
13 checks passed
@hanna-meda hanna-meda deleted the prototype/preload-adaptative-batch branch July 1, 2024 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module: preload type: enhancement Improvements that slightly enhance existing functionality and are fast to implement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[R&D] Adaptative batch size on Preload
7 participants