New Metric: Security vulnerability resolution Duration

[david-a-wheeler]

The "Closed Issue Resolution Duration" covers all issues, but is a little misleading. I routinely leave issues open for years if they are low importance, yet are nice-to-have - but that doesn't mean that the project is dead. What's missing is prioritization - if something is really important, then it should be resolved relatively quickly.

I propose "Security vulnerability resolution Duration" - the time it takes between when a vulnerability is reported to the project and when it is resolved. Security vulnerabilities have priority levels too, but clearly security vulnerabilities are more important to most projects than prettier formats or other nice-to-have things.

[david-a-wheeler]

You can partly counter the priorities of vulnerabilities the same way the CII Best Practices badge does: only measure this for vulnerabilities whose CVSS score is "medium or higher".

That would mean that this measure would show how quickly a project responds to medium or higher risk vulnerabilities. If a project won't respond in a timely way to vulnerability reports, then maybe users don't need know anything else about the project...!

[GeorgLink]

​I like the prioritization angle.

[germonprez]

Thanks @david-a-wheeler. Are you suggesting a new metrics aimed at vulnerabilities or are repositioning of this current metric?

I also think that we are not proposing any metric as an indicator of some value judgement (i.e., old open issues = dead project). We are simply putting the metric forward as a measure that could be investigated in more detail if the metrics users would like. Context is impossible in these cases so value judgements fail.

[david-a-wheeler]

I am proposing it as a new metric.

As far as value judgments go, I'm not proposing it as some sort of ethical failing. But if someone doesn't plan to do anything different depending on the value of a metric, don't see why they would care about a metric. Most metric frameworks that I'm familiar with stress that there is no point in measuring something unless there is a decision to be made by using those measures. Clearly a single measurement can be misleading, but that is why multiple measurements are useful.

[germonprez]

fair points :)

[GeorgLink]

I think we can solve this issue with a "filter" ... assuming that issues are tagged or otherwise identified by their "priority" or "security relatedness".