ci: solidity test benchmarks

[agostbiro]

This PR adds automated regression checks for the JS Solidity test runner. The setup is similar to how the RPC benchmark that benchmarks Hardhat Node-style workloads works.

Benchmark Suite

The benchmark runs the test harness of forge-std. This has two advantages:

• Largest coverage of cheatcodes in a single test harness which is helpful to catch regressions in all features.
• Makes sure our implementation works with forge-std which we intend to use.

The disadvantage is that forge-std is not representative of how users are writing Solidity tests. But this is not a concern as long as we’re not using the benchmark to guide optimizations, only to catch regressions.

There are multiple benchmark scenarios. The first one is called Total which measures the entire test harness execution of the forge-std test harness. The purpose of this scenario is to make sure that there are no regressions in the parallel test suite executor.

In addition to test suites, individual tests within test suites are also parallelized. This means that if we want to catch regressions in features covered by a certain test suite, we have to execute that test suite in isolation. For this reason, in addition to the Total scenario, we execute the following test suites as well:

• StdCheatsTest: environment-related cheatcode coverage + fuzzing.
• StdStorageTest: storage-related cheatcode coverage + fuzzing.
• StdMathTest: math-related utilities + it's very quick (~15ms on my machine), so regressions from FFI overhead should show up on this.
• StdUtilsForkTest: read-type programmatic fork tests.
• StdCheatsForkTest: write + fuzz tests for programmatic forking.

The rest of the test suites in forge-std test things implemented in Solidity or test file-system utilities, so they're less interesting for us.

Benchmark Metric

The metric for all scenarios is the wall clock time measured from JS in milliseconds. The measurement excludes loading files from disk as that's out of scope for EDR. If there is a 10% regression on any of the scenarios above, the benchmark job fails (same as the RPC benchmark).

Baseline

The baseline is executing the same tests with forge test. Instructions to run the baseline are in the crates/tools/js/benchmark/src/solidity-tests.ts file.

I tested manually that performance is identical to forge test on my machine, so we don't have to keep running the baseline.

Benchmark Jobs

There are two benchmark workflows in the feat/solidity-tests branch and three benchmark jobs:

• EDR Benchmarks:

  • Run JS scenario runner benchmark for Hardhat Node style workload
    • Stores results for main and compares PRs against against those results
    • Dashboard: https://nomic-foundation-automation.github.io/edr-benchmark-results/bench/
  • Run JS Solidity test runner benchmark
    • This is the new job
    • Stores results for feat/solidity-tests and compares PRs targeting feat/solidity-tests against those results
      • Once feat/solidity-tests is merged to main, we'll switch to storing results for main
    • Dashboard will be live here once this PR is merged: https://nomic-foundation-automation.github.io/edr-benchmark-results/soltests/

• EDR Benchmarks for feat/solidity-tests

  • Run JS scenario runner benchmark for Hardhat Node style workload
    • Stores results for feat/solidity-tests and compares PRs against against those results
    • Once feat/solidity-tests is merged to main, we'll remove this

I reorganized the workflow files a bit to share Rust compilation cache + start caching the RPC cache for the Solidity test runner benchmarks.

[changeset-bot]

⚠️ No Changeset found

Latest commit: e9a013b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR