Incorporate feedback from the Ethical ML workshops #20

anssiko · 2022-05-04T15:16:26Z

Update Operationalization: Putting the Principles into Practice
Add Register of Risks and Mitigations

Preview | Diff

- Update Operationalization: Putting the Principles into Practice - Add Register of Risks and Mitigations

anssiko · 2022-05-31T07:29:20Z

Gentle ping, PTAL @dontcallmedom :-)

Some areas to focus on, mostly presentational:

Proposals welcome on how to better represent Risks -> Possible Mitigations mapping. Maybe with some CSS tricks we could use a two-column layout for Risks and Possible Mitigations side-by-side (when screen estate allows)? Someone with proper CSS-fu would need to help with that. Or we could consider that future work.
"This bullet intentionally left blank" in Possible Mitigations. I was thinking of turning these empty bullets into links that open a new GH issue to encourage contributions.

Comments:

I'd prefer to keep the source format as simple as possible and let CSS handle presentational aspects, if possible. Complex tables are tedious to edit.
To help editing the lists and correlate Risks with Possible Mitigations in source, maybe instead of:

1.

1.

1.

We should use:

1.

2.

3.

dontcallmedom · 2022-05-31T07:40:11Z

index.bs


+So for example, if we consider the principle of “Fairness and non-dicrimination”, a risk might be that biases in training data lead to model predictions that are less accurate for particular groups, resulting in real-world harms (e.g. denial of services). A mitigation might be to test the training data and model properly for bias, or indeed it might be to not make access to essential services dependent on ML systems.
+
+To provide a more structured approach to this, we have developed a workshop format which you can adapt and use.


Suggested change

To provide a more structured approach to this, we have developed a workshop format which you can adapt and use.

To provide a more structured approach to this, we have developed a workshop format that can be adapted and used to help identify such risks & mitigations.

I wonder if the details about the workshop (until "it's worth noting" below) should be moved to an appendix - while very useful, it feels a bit out of place in the flow of the document.

Moved to appendix. Text changes integrated into 13d34dc

dontcallmedom · 2022-05-31T07:41:28Z

index.bs

+
+To provide a more structured approach to this, we have developed a workshop format which you can adapt and use.
+
+The workshop is in two parts (please make a copies for your own use):


Suggested change

The workshop is in two parts (please make a copies for your own use):

The workshop is articulated around two documents that are expected to be completed interactively by participants:

Integrated into 13d34dc

dontcallmedom · 2022-05-31T07:42:15Z

index.bs

+Part One - [Ethical thinking workshop](https://docs.google.com/document/d/1f_PcByjW8-zXbYWeEyOl-RpZ3SKapm_MWSaNZ9ZOi4c/edit?usp=sharing) - is about using the principles to generate and prioritize potential risks
+
+Part Two - [Ethical Risk Canvas](https://docs.google.com/document/d/1hTQnpWC5KC4qIJB9-Kkd46yMVgDuTCQlA3MffqtUbCI/edit?usp=sharing) - is about digging deeper into specific risks and thinking about who they might impact and how best to mitigate them.
+


we should probably post a PDF copy of these documents somewhere if only for archival sake.

dontcallmedom · 2022-05-31T07:49:41Z

index.bs

+
+# Register of Risks and Mitigations
+
+Note: The risk register is work in progress and welcomes further review and tidying up.


I'm happy to leave this for a future pull request, but I think there is indeed substantive work to harmonize the phrasing of the risks, with possibly merging / sorting some of them as well.

dontcallmedom · 2022-05-31T07:51:54Z

index.bs

+
+### Possible Mitigations
+
+1. Browser-assisted mechanisms to find out about the limitations and performance characteristics of ML models used in a Web app. This could build on an approach published in Model Cards for Model Reporting where making this report machine-discoverable would allow for the web browser to offer a more integrated user experience. Another transparency tool is the [Open Ethics Transparency Protocol](https://github.com/webmachinelearning/ethical-webmachinelearning/issues/6).


In terms of mapping mitigations to risks, I would use a naming scheme for risks (e.g. FND-R1 for risk 1 under Fairness and Non-Discrimination), and then references these names from the mitigations that address them.

Done in b00171e

Incorporate review feedback from Dom

- Creates dfns for all risks and mitigations - Also adds Bikeshed bikeshed includes for repeated notes/issue blocks

anssiko · 2022-06-02T13:15:23Z

Thanks @dontcallmedom for your comments, PR updated.

anssiko · 2022-06-09T06:04:42Z

Gentle ping, PTAL @dontcallmedom.

Your feedback has been addressed. I'd like to merge this after your approval so the WG can advance with its W3C Note publication plan.

anssiko · 2022-06-09T07:31:14Z

TY @dontcallmedom!

@anssiko

SHA: a023c87 Reason: push, by @anssiko Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

anssiko requested a review from dontcallmedom May 4, 2022 15:16

Incorporate feedback from the Ethical ML workshops

b5a3e28

- Update Operationalization: Putting the Principles into Practice - Add Register of Risks and Mitigations

anssiko force-pushed the workshop-feedback branch from d382eec to b5a3e28 Compare May 4, 2022 15:23

dontcallmedom reviewed May 31, 2022

View reviewed changes

anssiko added 3 commits June 2, 2022 12:20

Move Workshop Format and Templates to Appendix

13d34dc

Incorporate review feedback from Dom

Implement a naming scheme for risks to mitigations mapping

b00171e

- Creates dfns for all risks and mitigations - Also adds Bikeshed bikeshed includes for repeated notes/issue blocks

Add Bikeshed includes

4951f69

Update Bikeshed issue include

a272f56

dontcallmedom approved these changes Jun 9, 2022

View reviewed changes

anssiko merged commit a023c87 into main Jun 9, 2022

anssiko deleted the workshop-feedback branch June 9, 2022 07:31

github-actions bot added a commit that referenced this pull request Jun 9, 2022

Merge pull request #20 from webmachinelearning/workshop-feedback

c4e029b

SHA: a023c87 Reason: push, by @anssiko Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorporate feedback from the Ethical ML workshops #20

Incorporate feedback from the Ethical ML workshops #20

anssiko commented May 4, 2022 •

edited by pr-preview bot

Loading

anssiko commented May 31, 2022

dontcallmedom May 31, 2022

dontcallmedom May 31, 2022

anssiko Jun 2, 2022

dontcallmedom May 31, 2022

anssiko Jun 2, 2022

dontcallmedom May 31, 2022

anssiko Jun 2, 2022

dontcallmedom May 31, 2022

dontcallmedom May 31, 2022

anssiko Jun 2, 2022

anssiko commented Jun 2, 2022

anssiko commented Jun 9, 2022

anssiko commented Jun 9, 2022


		So for example, if we consider the principle of “Fairness and non-dicrimination”, a risk might be that biases in training data lead to model predictions that are less accurate for particular groups, resulting in real-world harms (e.g. denial of services). A mitigation might be to test the training data and model properly for bias, or indeed it might be to not make access to essential services dependent on ML systems.

		To provide a more structured approach to this, we have developed a workshop format which you can adapt and use.


		To provide a more structured approach to this, we have developed a workshop format which you can adapt and use.

		The workshop is in two parts (please make a copies for your own use):

	The workshop is in two parts (please make a copies for your own use):
	The workshop is articulated around two documents that are expected to be completed interactively by participants:

		Part One - [Ethical thinking workshop](https://docs.google.com/document/d/1f_PcByjW8-zXbYWeEyOl-RpZ3SKapm_MWSaNZ9ZOi4c/edit?usp=sharing) - is about using the principles to generate and prioritize potential risks

		Part Two - [Ethical Risk Canvas](https://docs.google.com/document/d/1hTQnpWC5KC4qIJB9-Kkd46yMVgDuTCQlA3MffqtUbCI/edit?usp=sharing) - is about digging deeper into specific risks and thinking about who they might impact and how best to mitigate them.


		# Register of Risks and Mitigations

		Note: The risk register is work in progress and welcomes further review and tidying up.


		### Possible Mitigations

		1. Browser-assisted mechanisms to find out about the limitations and performance characteristics of ML models used in a Web app. This could build on an approach published in Model Cards for Model Reporting where making this report machine-discoverable would allow for the web browser to offer a more integrated user experience. Another transparency tool is the [Open Ethics Transparency Protocol](https://github.com/webmachinelearning/ethical-webmachinelearning/issues/6).

Incorporate feedback from the Ethical ML workshops #20

Incorporate feedback from the Ethical ML workshops #20

Conversation

anssiko commented May 4, 2022 • edited by pr-preview bot Loading

anssiko commented May 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anssiko commented Jun 2, 2022

anssiko commented Jun 9, 2022

anssiko commented Jun 9, 2022

anssiko commented May 4, 2022 •

edited by pr-preview bot

Loading