[ENH] Implement proper Lift curve; keep Cumulative gains as an option #5075
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5075 +/- ##
=======================================
Coverage 84.74% 84.74%
=======================================
Files 286 286
Lines 60043 60061 +18
=======================================
+ Hits 50884 50900 +16
- Misses 9159 9161 +2 |
janezd
force-pushed
the
lift-curve-cummulative
branch
from
9212d09
to
46f4b6a
Compare
|
|
||
| graph_name = "plot" | ||
|
|
||
| YLabels = ("Lift", "TPR") |
There was a problem hiding this comment.
"TPR" used to be "TP Rate" and we still have "P Rate", so I suggest we use the same naming.
There was a problem hiding this comment.
This was unintentional. Fixed. Thanks for noticing.
|
Documentation is missing for the new widget look and functions. Not sure if you plan to add that as well or @ajdapretnar will help with that? |
lanzagar
changed the title
janezd
force-pushed
the
lift-curve-cummulative
branch
from
46f4b6a
to
6bf723e
Compare
|
@ajdapretnar, I've rewritten the documentation and I ask for stamping the picture and reviewing the text. |
| The **Lift curve** shows the relation between the number of instances which were predicted positive and those that are indeed positive and thus measures the performance of a chosen classifier against a random classifier. The graph is constructed with the cumulative number of cases (in descending order of probability) on the x-axis and the cumulative number of true positives on the y-axis. Lift curve is often used in segmenting the population, e.g., plotting the number of responding customers against the number of all customers contacted. You can also determine the optimal classifier and its threshold from the graph. | ||
| The **Lift curve** shows to curves for analysing the proportion of true positive data instances in relation to the classifier's threshold or the number of instances that we classify as positive. | ||
|
|
||
| Cummulative gains chart shows the proportion of true positive instances (for example, the number of clients who accept the offer) as a function of the number of positive instances (the number of clients contacted), assuming the the instances are ordered according to the models probability of being positive (e.g. ranking of clients). |
There was a problem hiding this comment.
according to the models --> according to the model's
There was a problem hiding this comment.
Also, cumulative is written with a single m.
There was a problem hiding this comment.
... which I occasionally, but seldom do.
| 2. If test results contain more than one classifier, the user can choose which curves she or he wants to see plotted. Click on a classifier to select or deselect the curve. | ||
| 3. *Show lift convex hull* plots a convex hull over lift curves for all classifiers (yellow curve). The curve shows the optimal classifier (or combination thereof) for each desired TP/P rate. | ||
| 4. Press *Save Image* if you want to save the created image to your computer in a .svg or .png format. | ||
| 5. Produce a report. | ||
| 6. 2-D pane with **P rate** (population) as x-axis and **TP rate** (true positives) as a y-axis. The diagonal line represents the behavior of a random classifier. Click and drag to move the pane and scroll in or out to zoom. Click on the "*A*" sign at the bottom left corner to realign the pane. | ||
| 6. A plot with **Lift** or **true positive rate** vs. **P rate**. The dashed line represents the behavior of a random classifier. |
There was a problem hiding this comment.
P rate or positive rate (it would make sense to explain this as in the previous part with Lift)
Also, the second sentence refers only to cumulative gains. I'd make this a bit more obvious.
There was a problem hiding this comment.
It does not. The dashed line also appears for lift, but it's horizontal, at 1. It is not always at the bottom; lift curve can go below 1.
There was a problem hiding this comment.
Oh, didn't notice it! 👀
|
|
||
| References | ||
| ---------- | ||
| The widgets that provide the right type of the signal needed by the **Lift Curve** (evaluation data) are [Test & Score](../evaluate/testandscore.md) and [Predictions](../evaluate/predictions.md). |
There was a problem hiding this comment.
The Predictions part is true only for labelled data. Perhaps make this clear?
There was a problem hiding this comment.
Yes, but Test & Score warns you about missing target variable, while Predictions doesn't. I don't know, I just think it would be clearer that way.
There was a problem hiding this comment.
I understand what you meant. But it's somehow clear that the data has to have a target variable, so I'd rather keep it short.
|
|
||
| Handouts of the University of Notre Dame on Data Mining - Lift Curve. Available [here](https://www3.nd.edu/~busiforc/handouts/DataMining/Lift%20Charts.html). | ||
| In the example below, we observe the lift curve and cummulative gain for the bank marketing data, where the classification goal is to predict whether the client will accept a term deposit offer based on his age, job, education, marital status and similar data. The data set is available in the Datasets widget. We run the learning algorithms in the Test Learners widget and send the results to Lift Curve. to see their performance against a random model. Of the two algorithms tested, logistic regression outperforms the naive Bayesian classifier. The curve tells us that by picking the first 20 % of clients as ranked by the model, we are going to hit four times more positive instances than by selecting a random sample with 20 % of clients. |
There was a problem hiding this comment.
results to Lift Curve. to see their performance against a random model.
? remove the full stop? Also, Test Learners is now Test and Score.
There was a problem hiding this comment.
Whatever I was reading after writing this documentation, it was not this documentation. Sorry that you had to check my typos.
There was a problem hiding this comment.
Tell me about it. I proofread my papers three times only for the second reader to find about 10 typos in them. 😒
There was a problem hiding this comment.
... with the first one being the fifth word of the abstract.
janezd
force-pushed
the
lift-curve-cummulative
branch
from
6bf723e
to
589216b
Compare
janezd
force-pushed
the
lift-curve-cummulative
branch
from
589216b
to
b84cc57
Compare
Issue
Fixes #5056. The widget showed cumulative gain curve instead of lift chart. Now it can do both.
Description of changes
The function used to use sklearn's roc curve. This PR has its own, simpler function modelled by the one that was used.
Includes