Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should CLI filters override filter from with_options? #344

Open
MisterDA opened this issue Feb 28, 2022 · 0 comments
Open

Should CLI filters override filter from with_options? #344

MisterDA opened this issue Feb 28, 2022 · 0 comments

Comments

@MisterDA
Copy link
Collaborator

I'm disabling a set of tests that should run e.g., on Windows, using the filter parameter from 'a with_options. I've discovered that this filter parameter is overridden by the CLI arguments, and I think that's surprising. If the test suite is itself disabling some of its test cases, maybe they shouldn't be run even at all. For instance:

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

(* Run it *)
let () =
  let open Alcotest in
  let filter ~name ~index =
    if name = "string-case" && index = 0 then `Skip
    else `Run
  in
  run ~filter "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
    ]
$ dune runtest
$ (cd _build/default && ./alcotest_test.exe test string-case)
Testing `Utils'.
This run has ID `8YC0XDXU'.

  [OK]          string-case            0   Lower case.
  [OK]          string-case            1   Capitalization.

Full test results in `~\Tarides\alcotest_test\_build\default\_build\_tests\8YC0XDXU.
Test Successful in 0.001s. 2 tests run.

IMO Lower case shouldn't be run. Arguably the user knows better and the cli overrides the code, but as a dev if I've disabled tests in the code, if I want to change that I'm going to change the code directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant