Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Widget for statistical comparison of models #3891

Closed
ghost opened this issue Jun 18, 2019 · 3 comments
Closed

Widget for statistical comparison of models #3891

ghost opened this issue Jun 18, 2019 · 3 comments
Assignees

Comments

@ghost
Copy link

ghost commented Jun 18, 2019

Orange version
Expected behavior

A widget should be added to compare model performances, using t-test, wilcoxon signed rank test etc.

Actual behavior

There is no currently available widget for statistical comparison of the model performances

Steps to reproduce the behavior
Additional info (worksheets, data, screenshots, ...)
@ghost
Copy link
Author

ghost commented Jun 18, 2019

At least, it should be possible to see all performance metrics in validation process. For instance, in 10-fold cross-validation, the program should show us the the metric for each fold.

@janezd
Copy link
Contributor

janezd commented Jun 19, 2019

I'd rather not promote (frequentist) null-hypothesis testing in Orange (or elsewhere). It is not only conceptually wrong per se, but also doesn't fit well into data mining workflow because constantly reformulating your hypotheses invalidates the tests' premises.

It would be nice though to have a modern Bayesian alternative. Essentially this: https://baycomp.readthedocs.io/en/latest/functions.html#single-data-set, for pairs of methods on single data set. We don't have to depend on baycomp for this, this function is just a Bayesian reinterpretation of the t-test.

@janezd janezd changed the title Need statistical comparison widget Widget for statistical comparison of models Jun 19, 2019
@janezd janezd self-assigned this Dec 8, 2019
@janezd
Copy link
Contributor

janezd commented Jan 24, 2020

Implemented via #4261.

@janezd janezd closed this as completed Jan 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant