# Fairlearn: Sociotechnical "talking points"
Since Fairlearn does not have many active users, and does not yet have significant adoption in any particular domain or context, the [theory of change](https://hackmd.io/EBh01XPtRZGHEg76oD1rGw) implies there's an opportunity to broaden the kind of work we're doing.

We've also found that sociotechnical discussions and meetings have been great for building community and sparking deeper kind of conversation and engagement than many other open source projects related to fairness. See [Fairlearn: Seed ideas for scenarios](https://hackmd.io/gD8dwdPSRsqH3BxBt9terg?both) for that work, and for historical notes and links.

This is a strength to build on, but it's not as clear what this work looks like or what form it will take. But it will need to look different than what is commonly published in research papers (eg, metrics on a dataset, or with [open source fairness libraries](https://github.com/google-research/tensorflow_constrained_optimization/blob/master/examples/jupyter/Fairness_adult.ipynb)). In business terms, one way to think of this is that we're moving up the "value chain" to provide a higher-value service than just the implementation of the mitigation algorithm alone.

## What could this look like?
One prototype is mentioned in [Brainstorming: Website for "fairness as a sociotechnical challenge" community #498](https://github.com/fairlearn/fairlearn/issues/498). In this doc, we'll present a similar but smaller format that may work better for group discussion and collaborative design.

These "sociotechnical talking points" are not related to the Fairlearn Python library per se, but the design intention here is that this would be part of the broader Fairlearn project and its website, in line with the theory of change above.

One way to visualize these "talking points" is to imagine that a data scientist called the team, and asked for advice. If we knew and cared about that data scientist, we might not always be able to say "you should just never build this, and shut down your organization and build up a more inclusive and participatory approach to the whole thing." Saying that doesn't actually help make it happen. So in these "talking points," we're trying to discover if there is good advice we can give, so that in that situation, with limited power, agency and time, our data scientist friend has some better options than doing nothing altogether. SO another way to think about this kind of work is as examples of **a sociotechnical fairness consultant's "talking points"**.

This work should follow the same [Contributing example notebooks](https://fairlearn.github.io/contributor_guide/contributing_example_notebooks.html) points, and using a synthetic dataset is just fine.

### Adding multiple levels
We could also imagine different *levels* of "talking points" for different levels of power and influence. For example, in the same situation a higher-level set of talking points might focus on ways to have conversations with different stakeholders about how the system shift power, or might discuss how the labeling and classification methodology enforces a kind of "administrative violence" (Hanna et al. 2020). At the same time, a lower-level set of talking points might start with a library that can surface "quality of service" issues in system, or demonstrate how 'noise' in the sociotechnical system makes narrowly technical evaluations of accuracy less meaningful. This kind of approach would allow us to keep talking points focus on small actionable communication, while also directly speaking to the important of organizational and interpersonal dynamics in sociotechnical fairness work.

So for a single deployment context, each set of "talking points" would speak to different data scientists, since the personal identity of the practioners is a critical type of context. Practioners may also have different kinds of lived experiences that they might speak directly to. For example, when working on a predictive analytics system for higher education admissions, some practioners may be able to speak to their own experience in higher education admissions. Or when working on facial recognition systems, some black women practioners may be able to speak about how "quality of service" relates to their own personal experience with intersectionality in other contexts.

## A first draft format
Here's one sketch of what sociotechnical "talking points" might look like. The idea is that there are only three points, and that they aspire to meet the *implications* in *Theory of change* section. Additional points are italicized to show the thought process for developing these, and for discussing the prioritization involved.

The way we'd evaluate these as educational tools (or even conduct research on them!) is to imagine if practicing data scientists and developers who are new to fairness could take action on these "talking points" and that would add value to the project within an hour. See the [Theory of change](https://hackmd.io/EBh01XPtRZGHEg76oD1rGw) doc for more on why starting with *small bites* and *actionable communication* is important.

## Draft examples: sociotechnical "talking points"
### Example #1: Identifying potential tax fraud
> You're a member of an analytics team in a European country, and brought in to consult about a project that has already started to scale the deployment of models for predicting which tax returns may require further investigation for fraud. The team has used a model trained in other jurisdictions by a large predictive analytics supplier, and hopes that they can leverage this at a lower cost that would be required to invest in the capability in-house. [Veale et al. (2018)](https://arxiv.org/pdf/1802.01029.pdf)

| Talking points <img src="https://avatars0.githubusercontent.com/u/1056957?s=400&v=4" width="32" /> @kevinrobinson 7/29/20
| --- |
| Clarify the goals of the system. Broadly speaking, the goal may be revenue maximisation or deterrence. Quantifying deterrence is obviously tricky, so further clarification will be needed. When doing so, remember that if groups targetted for investigation can be 'othered' in some way, then there may be minimal deterrence of the broader population. Even revenue maximisation has subtleties, since tax systems are complicated. It is possible that some instances of tax fraud may be recovered from other actors 'downstream' (a very simple example: if a person buys things with money from evaded income taxes, they are still going to pay sales taxes on the things they buy). There is an issue of fairness with respect to _who_ is paying the taxes, but the overall tax revenue may be unchanged.
| Seek to have explainable models, whether by using a glassbox model, or applying an explainer to an opaque model. Ensure that these explanations are passed on to the human investigators.
| Create a simulation showing how small changes in the dataset within your jurisdiction produce different predictions from the third party model, and the downstream implications.
| Express harms to people that are selected for investigation in terms of time and money, particularly for false positives (if there is a way to understand this). Make tangible the harms of fraud to other citizens in terms of reduced funding or quality of service for specific government services. If funding is prioritized, look at those specific services and consider asking citizens using those services for their perspective on how reduced funding would impact those services.

Other actionable steps:
- Ask about the capacity to investigate potential fraud, and learn more about that investigation process separate from the ML prediction system (eg, selection rate, but also the amount recouped from investigations). Ask people on the investigation team how such a system would help, and what outputs they require (estimates of fraud probability and recoverable amounts might be more useful than a binary classification)
- If the motiviation for the project is cost savings, include the cost of developing, auditing, and maintaining the system (which will include getting updates to the model from the external supplier and subsequent revalidation work) in any cost saving projections.
- Run audits on different demographic characteristics that are available.

### Example #2: Predictive grading in UK higher education admissions
> From discussion in https://github.com/marielledado/fairlearn-student-performance/blob/master/notebooks/analysis.ipynb, more abstractly than a specific concrete deployment context.

| Talking points <img src="https://avatars1.githubusercontent.com/u/873454?s=460&u=0aa11e92d10d4ec67c4275f33359c142f549203d&v=4" width="32" /> @hannawallach 7/23/20 |
| --- |
| I'll write up a paragraph about my own lived experience in the UK system, and you can share that with your team to humanize and personalize the kinds of harms that the system can lead to.
| Start a conversation with your team about the ways that grading and school systems in general can function as systems of large-scale oppression.

| Talking points @arjunsingh on 7/23/20 |
| --- |
| Frame the cost of inaccurate predictions in terms of the overall benefits. While all systems will have some level of error, and while there will always be outlier data points in any dataset, this doesn't mean we should never build systems. So when discussing harms that stakeholders may be sensitive to, make sure to frame them in terms of the benefits of the system as well (eg, in terms of how many students are receiving access to services that wouldn't otherwise be able to).

### Example #3: Customer Service triage

Philip works at a technical consulting company (Company A). One of the services his company provides is setting up an single mailbox to route incoming emails. Philip is working with a client who provides HR services (Company B) to create an email routing system. Company B works with hundreds of other companies to provide HR services related to employee insurance, well-being, payroll, personnel and training. So it's common for HR employees in downstream companies to email Company B for help with navigating specific HR issues within their own companies or for their own employees. Philip's role over the next month is to create a classification system for labeling emails in one of 32 categories. The output of that system is then used to route the email to the point of contact assigned to each category, which is maintained in a separate system. To start the classification work, the company previously set up a party keyword extraction system that can extract ~1000 binary features from an email. The company considers this keyword extract system "finished," so revising it or setting up a recurring service contract is unlikely unless it would lead to a significant change in customer feedback through Net Promoter Score surveys. [consulting blog post (2020)](https://customers.microsoft.com/en-us/story/774221-securex-professional-services-m365)

| Talking points |
| --- |
| People whose word choices lie outside those used to train the keyword extraction system could suffer two types of harms. 1) Loss of privacy due to incorrect routing of emails; and 2) Delay in service due to incorrect routing
| Since these emails are composed by HR professionals in the downstream companies, they _should_ be sufficiently anonymised. However, to minimise the chance of problems, identify particularly sensitive categories (such as harassment) and increase the weighting of the corresponding training examples. It might even be helpful to have the categories rated for sensitivity on a scale of 1-10, and use those as weights on the training examples.
| Delayed responses could cause the employees who ultimately originated the inquiries to feel unvalued (especially if the they are raising questions about discrimination or similar issues). Incorrectly routed messages will have to be manually redirected, so establish a process to minimise the time penalty. A simple way to do this would be to instruct workers to prioritise redirected messages. As part of this process, it is essential to define what 'timely' means for trouting messages to the correct recipient. Unfortunately, the greatest delay is likely to be the time taken for the receipient to compose their response; all the routing system can do is minimise any extra delays.

Other Actionable Steps:

- Establish a process for regularly retraining the classification system using misdirected messages as key examples
- If incorrect routing is causing trouble in customer feedback, and retraining the classifier is having minimal effect, then analyse the keyword extraction system. Pay special attention to differences in the keywords extracted from correctly and incorrectly routed messages. These could provide the evidence needed to reopen the matter of the keyword extraction system