# Challenge 3 – Misinformation
Office for Statistics Regulation
Ed Humpherson, Director General for Regulation
Kirsty Garratt, Head of Private Office and Casework Manager
Suzanne Halls, Head of Public Affairs and External Relations
Helen Miller-Bakewell, Head of Development and Impact

Ann Smith [e-mail]
Archie Herrick [e-mail] - [e-mail]
Robert MacKay [e-mail]
Marie Oldfield [e-mail]
Vincent Jansen [e-mail]
Tom King [e-mail]

# Misinformation Objectives:
Define misinformation and consider usable terminology for subsequent discussions.
Indentify desirable publication protocols and salient features of reliable communications.
Consider strategies for assessing cases presented to the Office for Statistical Regulation (OSR)

# Potential follow on investigations:
Identify key words for assessment of validity.

# Introduction

# Background Information OSR

Annual Review of UK Statistics Authority Casework 2020/21
Published:
30 September 2021
Last updated:5 October 2021

In the period 1 April 2020 to 31 March 2021 the Authority considered 323 pieces of casework. Nearly three times the number in the previous year (109 cases).

Health and Social Care made up 72% of all cases, driven by the COVID-19 pandemic.

Internally-generated casework accounted for 16%, a smaller proportion than in 2019/20.

76% of cases looked into this year were related to the pandemic in some form.

48% of cases related to quality, reliability and trustworthiness of statistics. The first time this had been the most common category.

The average (median) time taken from opening to closing a case was 10 days (mean 15 days). Compared with 13 (median) and 20 days (mean) in 2019/20.

# The Challenge:
How can we distinguish between...

...misleading uses of statistics, data and evidence

vs

legitimate alternative interpretations of the same underlying evidence?

AS Reframe as objectives:
Can we assess prior cases and categorise features of misleading and non misleading cases.
>levels of 'offense'

Transparancy of methods and data availability.
Critical thinking skills

Code of practice for statistics ed 2.1 Revised 5th May 2022
https://code.statisticsauthority.gov.uk/working-in-line-with-the-code/how-the-code-of-practice-for-statistics-can-help-support-evaluation/

Three pillars
>Trustworthyness (truthful, impartial and independent)
>Quality
>Value

More detailed advice on application of TQV when conducting an evaluation is given in The Magenta Book, HM Treasury guidance on what to consider when designing an evaluation. Advise is structured in the use, design and impact of evaluation with emphasis on transparency ahnd communication of information in a useable format.

1. Ambitions for the use of evaluation
A culture of evaluation supports effective policy and relies on openness: being open to learning what works and what doesn’t, as well as open about the use of evaluation and releasing information in an orderly way so others can use it:

publish information about the evaluation ahead of doing it
publish the results of all evaluations in a managed way
publish the data and methods so others can scrutinise and replicate the analysis
The Trustworthiness pillar can help you and your organisation in being open.

Trustworthiness: confidence in the people and organisations that produce statistics and data

Trustworthiness results when the people, systems and processes within organisations enable and support the production of data and analysis.

It will come from the organisations being well led, well managed and open, and the people who work there being impartial and skilled in what they do.

Showing you are trustworthy means being truthful, impartial, and independent – free from vested interests.

Information is shared responsibly and with integrity – being open and transparent about your decisions, plans and progress.

It means you look after people’s information ethically and securely, manage data in ways that are consistent with relevant legislation, and serve the public good.

2. Ambitions for the design of evaluation
Design choices lie at the heart of excellent evaluation. High-quality evaluation happens when:

the evaluation question is well defined
the design is proportionate for its purpose and the scale of the policy
the right data and robust methods are selected, tested, and explained
The Quality pillar can help you in making these design choices.

Quality: data and methods that produce assured statistics

Quality means that statistics meet their intended uses, are based on suitable data and methods, and are not materially misleading.

Quality requires skilled professional judgement about collecting, preparing, analysing, and publishing statistics and data in ways that meet the needs of people who want to use the analysis.

The organisation should communicate the quality of the data and methods chosen to its audiences.

A commitment to quality is shown by considering and describing how the evaluation sources and selects data and chooses and tests methods to ensure their suitability. It is reflected in seeking expert review at key stages and by informing users about the quality and limitations of the evaluation.

3. Ambitions for the impact of evaluation
Impactful evaluation addresses the right questions and ensures the correct understanding of the evidence. It means that:

there is a clear basis for the evaluation informed by stakeholder insight
the evaluation clearly finds at an early enough point whether the policy is beneficial
the findings are clearly and accessibly communicated with the audiences in mind
The Value pillar can support you in ensuring your evaluation is impactful.

Value: statistics that support society’s needs for information

Value means that the analysis is useful, easy to access, relevant, and supports understanding of important issues by effectively communicating evidence as required by various audiences.

The users and the intended use should be at the centre of the evaluation, by understanding the research question and acting on the insight.

Published data and analysis are made equally available to all, being open about the nature of the evaluation and how it serves the public good. It enables its replication and reuse by others.

## Definitions

Definition of trustworthy information:

What key elements indicate validity?
Presenting truthful alternatives rahter than a persuasive argument.
How to educate, without controlling, strong priors?

Can we quantify evidence? eg. give probabilistic values to phrases/ sources...?

We can quantify how good one explanation is vs another. What we cannot do is say what the truth is as that would require sifting through all possible explanations. Therefore finding the best explanation requires the subjective choice of choosing a set of possible alternatives. In my view there is no strictly objective method of finding the truth. We can find better explanations, but unless there is a finite set to choose from, we fundamentally cannot know if we have the right answer (this is largely inspired by the narrative of Burnham and Anderson, 2002)

Is this asking for distinguishing legitimate interpretations from illegitimate interpretations? Aside from factual errors, the distinction is subjective as it refers to a framework of legitimacy, it seems to me. If the challenge is asking for an objective distinction, I doubt if this is possible, or desirable.

# How do we decide if something is misleading?

Misleading’ is used to cover a wide range of situations

Sometimes, the judgement we are being asked to make revolves around the merits of the argument that the user is making, rather than the use of statistics in itself.

We have been thinking about the idea of misleadingness – what it is, and how we should approach it in our work.

“We are concerned when, on a question of significant public interest, the way statistics are used is likely to leave audiences believing something which the relevant statistical evidence would not support.”
https://osr.statisticsauthority.gov.uk/publication/misleadingness-a-short-thinkpiece/

# Judging misleadingness

Here are three potential approaches to judging misleadingness:

1: **Materiality and intention** – an approach which focuses on the significance of the statement being made. What were the intentions of the speaker?

2: **Audience** – an approach which focuses on audience understanding. Were the audience misled about what the statistics were telling them?

3: **Case-based** – an approach which focuses on particular features of the presentation of statistics. Is the style of presentation unclear and likely to mislead?

“To ensure users can confidently make decisions about the statistics that are presented to them, using them without question to access what they require and need”.

A simple way to achieve these outcomes is to ask the right questions about the statistics within your communications messages.

Improving statistical literacy for both writer and reader is imperative to ensure each has the necessary skills to assess the validity of publications. An initial assessment might begin with asking the right questions.

Does it look right – is it an implausible number?
If it’s unusual it could be wrong – what’s behind the surprise?
What exactly are we measuring, and why?
Where do the data come from – what is the backstory?
Be curious about someone else’s data – could a little more research help me understand this better?
How are the statistics calculated – is the source reputable and did they show their working?
Only compare the comparable – watch out for changes in definitions and different ways to measure the same thing
What led to what – is this really a causal relationship?
Understand the assumptions – has anything been left out? What happens if they are wrong?
How sure are you about this – numbers contain uncertainty – how precise are yours?
Put things in context – what’s the historical trend?
Presentation is key – good use of pictures and graphics help convey meaning and should never cause confusion or misrepresentation

# Can we identify a protocol for judging validity?
Key factors, TQV, in assessment of information taken directly from the statistics code of practice and the magenta book. Is the information from a trustworthy source of good quality and does it add public value?

Onora O'Neill defined 'intelligent transparency' against 4 criteria:
* accessibility
* assessability
* useability
* intelligibility

The second would be particularly important for evaluating the validity.

# Avoiding misleading statements:
'Doubles your chances of infection'
Absolute vs Relative risk.

Summarising and communicating in non-manipulative way. Whether that's verbal or visual communication.

Informing not influencing attitudes, emotions or attempting to convince them of a particular perspective. To present the facts and be as truthful as possible.
Not always a balanced argument if evidence not balanced but a fair representation of the facts.

Easier said than done! Babbage and co on setting up the Royal Statistical Society (1834) had this goal-facts only; figures and tables no personal or biased opinions...but what are the facts without conjecture? Context needed, latent features identified etc. (Spiegelhalter)

## Logical Fallacies & Statistical Biases
There are many classic errors made in communicating about statistics, such as implying an observed association is evidence of a specific cause rather than potential confounding. Would it be useful to have a list of these, especially things like the ecological fallacy that are commonly made inadvertently? Similarly, although there are many claims that things are biased, the technical definition, of a difference between the expected value of an estimator and the true value of a parameter is important. Selection, survivorship, and others all produce misleading statements, which can be propagated by people who are unfamiliar with statistics.

## Statistical Literacy
One was OSR could combat misinformation would be guidance on particular types of errors, or such information they could point people to when they see it. They could also expect producers to explain these limitations, especially if inadvertently misleading statements are made based on official statistics. Where the intentionality of the statement is naive, then the response should be supportive as well as preventative and corrective.

# Increasing public engagement and Interest
OSR seen rise in caseload over the past two years from approximately 150 to 300 per annum.
Fuelled by pandemic bombardment (data and opinion wise).

Also huge increase in reporting across society on widening number of social platforms.

How can measures akin to rigeurs of academic publication be introduced without taking away the dynamic discussion or narrowing participation?

What is misinformation?

- 1 # The online information environment

https://royalsociety.org/topics-policy/projects/online-information-environment/

"Within this report, ‘scientific misinformation’ is defined as information which is presented as factually true but directly counters, or is refuted by, established scientific consensus. This usage includes concepts such as ‘disinformation’ which relates to the deliberate sharing of misinformation content."

I challenge this, because it regards established scientific consensus as the truth. I agree(Marie)

- 2 FUN-MOOC on Critical thinking: data and fallacies https://lms.fun-mooc.fr/courses/course-v1:CY+156007+session01/

- 3 Turing programme on Understanding vulnerability to online misinformation
https://www.turing.ac.uk/research/publications/understanding-vulnerability-online-misinformation

- 4 Relevance to Online Safety Bill.

## UK Government's Online Safety bill
From UK Goverments Fact-Sheet
"The duty of care will require platforms to have robust and proportionate measures to deal with harms that could cause significant physical or psychological harm to children, such as misinformation and disinformation about vaccines. Platforms will also need to address in their terms of service how they will treat named categories of content which are harmful to adults, likely to include disinformation. This will mean:

- all companies will need to remove illegal disinformation, for example where this contains direct incitement to violence
- services accessed by children will need to protect underage users from harmful disinformation
- services with the largest audiences and a range of high risk features (Category 1 services) will be required to set out clear policies on harmful disinformation accessed by adults
- The regulatory framework will also include additional measures to address disinformation, including provisions to boost audience resilience through empowering users with the critical thinking skills they need to spot online falsehoods, giving Ofcom the tools it needs to understand how effectively false information is being addressed through transparency reports, and supporting research on misinformation and disinformation.

## How social media platforms define and deal with misinformation
### Issues with working definitions

*Do they need to, is it their role, is it open to abuse, does it include regulation and policies i.e internal government published papers banned in 2020,21 due to going against 'public health policy' as dictated by social media platforms and academics banned

### Meta's 'False news' policy
https://www.facebook.com/formedia/blog/working-to-stop-misinformation-and-false-news
https://en-gb.facebook.com/business/help/2593586717571940
- No discernable attempt to clearly define 'false news'
- 'false news' is identified using user behaviour and through fact-checkers
-- Generally not removed, but 'ranked lower' on a newsfeed
-- For example, if reading an article makes people significantly less likely to share it, that may be a sign that a story has misled people in some way
-- Fact checkers are "community and third party"
- 'Repeat offenders' may have their ability to monitise, advversites or register a news page removed
-- Content that is 'near-identical' to 'false news' is automatically included

- case study of 'French language' video on hydroxychloroquine and azithromycin
-- The board overturned the decision to remove a video on the drugs, in part because the drugs mentioned are prescription drugs in France, which would require individuals seeking them to interact with a physician.
### Twitter's 'misleading information' policy
https://help.twitter.com/en/resources/addressing-misleading-info
https://help.twitter.com/en/rules-and-policies/medical-misinformation-policy
https://help.twitter.com/en/rules-and-policies/manipulated-media
- 'misleading media' is defined as 'synthetic, manipulated, or out-of-context media that may deceive or confuse people and lead to harm'
- For misleading media to be removed, it must:

1) **Include media that is significantly and deceptively altered, manipulated, or fabricated,**
-- Whether media have been substantially edited or post-processed in a manner that fundamentally alters their composition, sequence, timing, or framing and distorts their meaning;
-- Whether there are any visual or auditory information (such as new video frames, overdubbed audio, or modified subtitles) that has been added, edited, or removed that fundamentally changes the understanding, meaning, or context of the media;
-- Whether media have been created, edited, or post-processed with enhancements or use of filters that fundamentally changes the understanding, meaning, or context of the content;
-- Whether media depicting a real person have been fabricated or simulated, especially through use of artificial intelligence algorithms

2) **Include media that is shared in a deceptive manner or with false context**
-- Whether inauthentic, fictional, or produced media are presented or being endorsed as fact or reality, including produced or staged works, reenactments, or exhibitions portrayed as actual events;
-- Whether media are presented with false or misleading context surrounding the source, location, time, or authenticity of the media;
-- Whether media are presented with false or misleading context surrounding the identity of the individuals or entities visually depicted in the media;
-- Whether media are presented with misstatements or misquotations of what is being said or presented with fabricated claims of fact of what is being depicted

3) **Include media likely to result in widespread confusion on public issues, impact public safety, or cause serious harm**
--Threats to physical safety of a person or group
--Incitement of abusive behavior to a person or group
--Risk of mass violence or widespread civil unrest
--Risk of impeding or complicating provision of public services, protection efforts, or emergency response
--Threats to the privacy or to the ability of a person or group to freely express themselves or participate in civic events, such as:
a.Stalking or unwanted and obsessive attention
b.Targeted content that aims to harass, intimidate, or silence someone else's voice
Voter suppression or intimidation

- In the absence of other policy violations, the following are generally not in violation of this policy:
- Memes or satire, provided these do not cause significant confusion about the authenticity of the media;
- Animations, illustrations, and cartoons, provided these do not cause significant confusion about the authenticity of the media.
- Commentary, reviews, opinions, and/or reactions. Sharing media with edits that only add commentary, reviews, opinions, or reactions allows for further debate and discourse relating to various issues and are not in violation of this policy.
- Counterspeech. We allow for direct responses to misleading information which seek to undermine its impact by correcting the record, amplifying credible information, and educating the wider community about the prevalence and dynamics of misleading information.
- Doctored or fake Tweets, social media posts, or chat messages

### EU law attempts to define misinformation
https://policyreview.info/articles/analysis/perils-legally-defining-disinformation

**Common components of definitions:**
(i) factual or misleading nature of the information,
(ii) harm,
(iii) intention of the actor,
(iv) economic gain,
(v) strategic dissemination.

*"When the definition of disinformation is explicitly discussed, the general consensus seems to be that there is no clear, uniform or legal definition (“Joint Declaration on Freedom of Expression”, 2020; Tambini, 2020; Van Hoboken et al., 2019; Nyakas et al., 2018)."*

*‘Mis-information is when false information is shared, but no harm is meant. Dis-information is when false information is knowingly shared to cause harm. Mal-information is when genuine information is shared to cause harm, often by moving information designed to stay private into the public sphere’ (Wardle & Derakhshan, 2017, p. 20*

**EU High Level Expert Group Definition***‘false, inaccurate, or misleading information designed, presented and promoted to intentionally cause public harm or for profit’ (HLEG, 2018, p. 10)*

**European Comission Definition***‘verifiably false or misleading information that is created, presented and disseminated for economic gain or to intentionally deceive the public, and may cause public harm’ which is understood to be ‘threats to democratic political and policymaking processes as well as public goods such as the protection of EU citizens' health, the environment or security’ (2018a, s. 2.1).*

Lithuania has an explicit statuatory prohibition on disinformation, defined as:
""*intentionally disseminated false information’ (Art. 2). As such, this aligns with Wardle and Derakhshan’s definition, in that the definition contains the elements that (a) the information must be false, (b) there must be a specific intention; and (c) causes certain harms. However, it is limited to causing harm to a specific person, and does not include public harm; while there is no requirement of economic gain (as envisaged in the EC’s definition). Further, the focus is on the dissemination of disinformation, and not the creation.*""
**
The four international special mandates on freedom of expression have stated that laws containing prohibitions on dissemination of ‘false news’, which are ‘vague and ambiguous’, are ‘incompatible’ with international standards on freedom of expression, and ‘should be abolished’ (Joint Declaration, 2020, s. 2(a

UN Special Rapporteur on freedom of expression has emphasised how the concept of disinformation is an ‘extraordinarily elusive concept to define in law’, and susceptible to providing executive authorities with ‘excessive discretion to determine what is disinformation, what is a mistake, what is truth’ (“Joint Declaration on Freedom of Expression”, 2020, para. 42)As such, the penalisation of disinformation is ‘disproportionate’ under international human rights law (“Joint Declaration on Freedom of Expression”, 2020, para. 42). Further, the UN Human Rights Committee has found that prosecution for the ‘crime of publication of false news’ on the ground that the news was false, is in ‘clear violation’ of the right to freedom of expression (Human Rights Committee, 1999, para. 24).*

## OSR Policy on transparency
*"We monitor the use of data and investigate concerns where we see unpublished figures – such as management information or models – being used publicly to inform Parliaments, the media and the public. In deciding whether to intervene we consider whether:

• equality of access has been preserved
• data quoted are material to public debate
• data are being used to justify important government decisions
• there is appropriate explanation of context and sources, including being clear about caveats and quality concerns
• figures form part of a coherent narrative across different sources of information

#OSR Policy for intervention
- Transparency and assessability is central to the decision to intevene, with intervention if:
1) official statistics are shared before publication
2) the advice of professional statisticians is ignored
3)

How do in and out groups contribute or get pushed towards misinformation and if this happens really how much of a threat is it

How can we educate users in critical thinking to be able to sift and find information in the bottomless pit that is the internet"*

How is freedom of speech and expression affected by these terms. RSM: I agree this is a big concern. We're heading for a ministry of truth scenario. And here's what is happening in the USA: https://www.hstoday.us/federal-pages/dhs/dhs-standing-up-disinformation-governance-board-led-by-information-warfare-expert/

VJ: reading the above, on the one hand these the word baby and bathwater spring to mind: for instance, if we label information that confuses people as misinformation, an awful lot of information will fall into that net that shouldn't. On the other hand, information can be weaponised and abused, and therefore having safeguards in place makes sense.

When do these terms become weaponised and are they appropriate in a commerical environment

Paternalistic society currently 'keeping people safe' i.e feeding into a fear machanism under false pretences and modifying behaviour - has politics gone too far in trying to force change in their desired direction (every action has many opposite reaction - some we cant imagine until we implement things)

Why do we need these terms, are they appropriate or neccessary? concept / motivation behind them?

How to identify and report fake news items BBC
https://www.bbc.co.uk/news/38053324

Things to ask yourself before you share a claim
In our earlier guide to spotting fakes during the US election, we gave you some ground rules to help with identifying false or misleading reports.

Have I heard of the publisher before?
Is this the source I think it is, or does it sound a bit like them?
Can I point to where this happened on a map?
Has this been reported anywhere else?
Is there more than one piece of evidence for this claim?
Could this be something else?
# visuals

some visuals, should that be handy

Great moon hoax (early disinformation example https://en.wikipedia.org/wiki/Great_Moon_Hoax)

the end is nigh (Tintin, shooting star)
you can't have enough Tintin
![](
https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fen.tintin.com%2Fimages%2Fjournal%2Fjournal%2F00388%2FC0905D3en.jpg&f=1&nofb=1)
Operation Infektion (KGB disinformation campaign (https://en.wikipedia.org/wiki/Operation_INFEKTION )

# Factors
On the challenge to identify criteria some possibility for what would be characteristic of a legitimate claim:
* valid
* contextualised
* derivative

The **valid** claim would be about being supported by the underlying evidence in terms of not being fallacious. Where this is about the future it would require explicit conditions rather than absolute truth.
The **contextualised** claim would be reporting the population or counterfactual relating to the claim. For something about the future, this would include reference to credible alternatives.
The **derivative** claim would be based only on the evidence, so that speculation or extrapolation is clearly separated. For something about the future, it would include the uncertainty with which the evidence supports the claim.

The point is not to restrict what can be said but to separate statements that are legitimate from evidence, and other political speech which may be speculative, rhetorical or otherwise hyperbolic. As claims are derived from evidence, there will be limits to the claim, which can be related back to the evidence used. Further, the legitimacy of the claim is dependent on evidence but its legitimacy will only be verified if it is available in the extent envisaged as transparent.

As examples, claims about behaviour of individuals would be supported by evidence about individuals rather than simply populations. Claims about processes cross-sectionally would be in the context of the full extent of the process. Claims about the impact of a policy would be restricted to outcomes that were obtained rather than what they were proxy or surrogate for. The aim is that a claim is coherent and stands in relation to evidence, but does not exclude further discussion, indeed the limits of context, validity and derivation ought to be the focus of further attention.

# Information – misinformation, towards a formal definition? (from vincent)

A rather grand perspective on information and misinformation. I tried to write down some of the ideas I talked about during the workshop.

As a starting point I have used the some ideas on biological information (https://plato.stanford.edu/entries/information-biological/) and used notions from information theory, in a rather loose way.

As humans, or animals, we have often no inherent information about the state of the world. To find out what is like we need to make observations, gather information, and use this to learn about the world.

A way to formalise this process is through learning by means of Bayesian updating. The basic idea is that we start with a prior view of what the world is like. As we gather information we adjust our view of the world in accordance with the observations we make. Initially, our world view would have been very uncertain, and that I actually know very little for sure. For instance, if I need to fish for food, I need to know when is the best time to fish. Without ever having been to the river, I could guess that it makes no difference whether I fish at dusk, dawn or noon. But after a number of fishing trips I could update that view, as I tended to catch more fish at dawn and dusk, and less at noon.

The fishing trips are a way to explore, and personally gather information about the world. (This is a very loose description of Bayesian updating and can be used to mathematically describe the amount of information.)

Information, and learning, need not only come from personal information, it is possible that I gain information from others. If, through some form of communication, I can get other's views or observations, I can use such social information to update my world view and learn about the world.

Social information differs from personal information in that I have assurance that personal information is based on a direct observation of the world I want to learn about. With social information this is not necessarily the case: I do not really know were that information is coming from. Firstly, social information will be processed by whoever collected the information in the first place. Social information can be affected by the prior views of the person who transmitted the information and can be filtered in the process of passing on. Secondly, it is possible that the information that is passed was incorrect, possibly intentionally so. For instance, a fellow fisherman might emphasize the merits of midday fishing to protect his own catch. For either, the usefulness of social information can be affected and one should take care in using it. (there is a link to trust here)

But how do we judge or how do we quantify the usefulness of information? There is no signal in the information itself that tells us the value. The only way to tell whether the information is good or bad is by telling if it increases decreases my knowledge of the state of the world. If I know less what true state of the world is after I have used the information that I have received, that information was not helpful

This allows for a definition of the value of information, and thereby, misinformation: if I know the state of the world better after receiving the information, the value of the information is how much more I know. If the information had a positive value it is helpful, if know less what the state of the world is the information was not helpful, and could thereby be seen as misinformation. (Also here, think one can make this notion in a mathematical way.) **Misinformation could thus be defined as information that leads me away from the truth.** (wikipedia also worth looking at
https://en.wikipedia.org/wiki/Misinformation)

But herein lies a problem. As an individual, I do not know what the exact state of the world is, the truth if you like, and therefore I cannot judge the value of information on this basis as the value is measured by using a touchstone which I do not have access to[^first].

Tom King added: There is another way of thinking about this, which comes from science advice, where advisory approaches are categorised in 4 types (Pielke):
* Honest Broker
* Pure Scientist
* Issue Advocate
* Science Arbiter

In broad terms, the arbiter waits to be asked, the advocate pushes evidence in support of their concern, while the broker tries to anticipate. Perhaps the first is trying not to mislead, the second is trying to lead, and the third is trying be impartial (the pure scientist stays above the fray). But these are all operating on the basis that there is a truth, which we may lack complete information about, but also need guidance interpreting it.--

The same problem extends to a definition of misinformation: if misinformation is information that is not reflective of the truth, we can only judge this if we know exactly what the truth is. And, mostly, we do not know what the truth is with certainty.

A very grand, perspective, but does it help? By thinking in such a framework few thoughts that came to me that are perhaps more applicable and practically useful:

- Judging if something is misinformation or not needs reference to a truth. Only in rare cases will this be possible, for instance, if the information pertains to an specific event or specific data that we have full knowledge about. Any legal definitions are likely to be restricted to such cases only, but that doesn’t mean that it covers all unhelpful information and will leave cases of misinformation out of view that are not helpful in forming a truthful opinion. Whether it is useful to use the word “misinformation” is a moot point as the word “untruthful” is equally applicable such cases.

- Whether or not the information is misinformation is subjective in a definition of this kind. Perhaps my view of the world was so off, or so badly informed that the new information didn’t make it any worse.

- Misinformation is not always wrong or in conflict with the true state of the world. It is possible that my fellow fisherman feeds me selective information about fish that he caught in the middle of the day only. That is not necessarily wrong or a lie -- he might well have caught those fish at those times –- but it is unhelpful to me because it skews my view of the world away from what it really is.

- Judging information against the belief-system of a group or collective is not a way out if this dilemma: even if many know more than one, it is not a given that a large group has access to the absolute truth or is even close to it. In fact, through echo chambers and bubbles it is quite possible that a group is firmly convinced fof the accuracy of their beliefs through the feedback received within a group, but that is not a guarantee it is actually truth. And examples were dissenting opinions in the end became commonly accepted are not hard to find. Indeed Galileo, Darwin and Planck probably fell into that category. An example within epidemiology is toxic shock syndrome and the claims by Tierno (I remember the story from Laurie Garrett’s book "The Coming Plague", needs checking).

More on toxic shock https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3238331/

- Misinformation is easily given without the intention to mislead: if I communicate my view of the world, which is, at least in part, based on my prior beliefs, or filter the information I give through my view of the world based on priors, I could very easily dispense information that is unhelpful to others. This would be misinformation for the receiver, but not for the sender.

- If misleading information is information given with the intention to mislead and is therefore not necessarily the same is misinformation. Misleading information does not necessarily require untruthful information: selectively feeding data to someone will mislead, without those data having to be fabricated or untrue, they are just not representative.

- Information given based on or influences by someone's prior beliefs, is neither dishonest or malicious. It might be incorrect, we might not agree with it, it could be bonkers, but that does not mean it was not given with truthful intention.

- The process of information spread and information copying naturally leads to misinformation: a rare event will often be much talked about and give the impression that it happens much more frequently than it actually happens. By incorporating this information into our belief-system we build up a false view of the world (do I remember correctly that crime reporting and the general perception how frequent crimes is one such example.)

- Perhaps a question we should ask if we need a formal and legal definition of misinformation. It seems to me that the notion has an inherent subjective component to it, it is misinformation to me, but perhaps not to you. And as we rarely have absolutely truth to fall back on, why would my information be misinformation and yours not? The Burnham and Anderson (2nd ed 2002) book as well as the recent book by Bergstrom and West (2021) call in a very different context for the need for critical thinking. Burnham and Anderson writ in the context fo chosing models and hypotheses: “Our science culture does not enough to regularly expect and enforce critical thinking”. Bergstrom and West’s book is entitled “Calling bullsh!t” and that nicely sums the content up (It is a serious book and worth reading. [Link to authors' website](https://www.callingbullshit.org "Link here") should you want to read a bit more). Perhaps their notion has more everyday currency than a formal and general definition of misinformation.

[^first]: Perhaps there are ways around this by using an abstract notion of truth as in Burnham and Anderson (2002), but I doubt if that helps in coming to a practical or legal definition. In Burnham and Anderson's jargon, one could ask if the distance of my internal model from the abstract truth, given the data I have objectively collected is higher or lower after incorporating the new information. (Roughly speaking, is the likelihood of internal model, given my data, higher after I use the new information?). For personal judgment, this useful and it is of course what we do in cases when something is too good to be true. For general use this is no good though, as it depends on a personal collection of data as a reference, and is therefore also subjective measure.

##Summary day 2

The Challenge we were set:
How can we distinguish between…
…misleading uses of statistics, data and evidence vs legitimate alternative interpretations of the same underlying evidence?
Can we assess prior cases and categorise features of misleading and non misleading cases.
- This led to a discussion about what misinformation is. In a wider sense, misinformation is information that leads someone away from the truth.
It can manifest itself through the presentation of wrong or fallacious information, misrepresentation, misleading information or blatant lies and untruths.
misinformation is serious and has severe consequences.
- this is a very wide definition of misinformation. A useful analogue is to see the world as a market place of ideas. If there is perfect information and rational decision making this should lead to the selection of the best ideas that captures the information in the world best. Misinformation is information that disrupts this market.
- In a perfect world that would work. But the world is not perfect. Social media, for instance, often do not work like a open market but consist of micromarkets, bubbles in which information echoes around, distorting the signal in the process. And players can manipulate this market through disinformation.

What can we do about misinformation
- Firstly, players on the market should avoiding giving misinformation of making inadvertently misleading statements, use good practice, Informing not influencing attitudes. To present the facts and be as truthful as possible give a fair representation of the facts.
But not some misinformation might not be inadvertently given.

- There is a place for regulation and intervention at govtal or platforms, in case of social media, but any policy or enforcable regulation needs to find a balance between limiting misinformation while not infringing on other rights, such as freedom of speech and expression.

We looked at policies of govt, social media platforms, EU
- Example, the EU has definition of misinformation, disinformation and malinformation and in some states, laws that prohibit these.
But the four international special mandates on freedom of expression have stated that laws containing prohibitions on dissemination of ‘false news’, which are ‘vague and ambiguous’, are ‘incompatible’ with international standards on freedom of expression, and ‘should be abolished’

- In terms of the remit and powers of the OSR they can act on misinformation if official statistics or data used in a document or statement are presented in such a way that they are likely to mislead the public or undermine the integrity of official statistics

However, this limits misinformation to cases where there is intent (misleading) and where the way information is used is demonstrably untrue. It needs to be verifiable that the information is not correct.
This covers part, but not all misinformation. Misinformation, for instance, can be based on accurate individual facts that are misrepresented.

It is neither likely nor desirable to regulate and prohibit against all misrepresentation.

- Beyond that, users of information – the general public – need to be critical and apply critical thinking to navigate in a world of which an unprecedent amount of information, but also misinformation, is present.
- There is a need for education on critical thinking. To allow the general public to exercise critical thinking, providers of data, and particular official providers of data to be open and transparent, make data and methodology available so that the data is accessible and can scrutinised and the analysis replicated (see code of practice for statistics

- The OSR asked for ideas on how to categorise features of misleading and non-misleading information, methodological help. For tomorrow, use of Bayesian model comparison.

JB: As promised, I looked into different types of misinformation that spread on social media. I found this useful [Twitter thread](https://twitter.com/i/events/859394302304235521?lang=en) and [accompanying article](https://firstdraftnews.org/articles/fake-news-complicated/) from Clare Wardle. She has developed a typology with an accompanying matrix (see below) which links what types of misinformation are spreading and the corresponding motivations for spreading each type. 
1. Satire or Parody: No intention to cause harm but has potential to fool
2. False Connection: when headlines, visuals or captions don’t support the content
3. Misleading Content: misleading use of information to frame an issue or individual
4. False Context: when genuine content is shared w/ false contextual information
5. Imposter Content: when genuine sources are impersonated
6. Manipulated Content: when genuine information or imagery is manipulated to deceive
7. Fabricated Content: new content that is predominately false, designed to deceive & do harm

# Resources:

How OSR secures change, Ed Humpherson, Director General for Regulation

https://osr.statisticsauthority.gov.uk/how-osr-secures-change/

Communication (Spiegelhalter)
https://www.socialsciencespace.com/2018/04/david-spiegelhalter-communicating-statistics/

Defining Ground Truth, Ed Humpherson
https://osr.statisticsauthority.gov.uk/ground-truth/

'How to decide what the truth is' Frank Kelly link?
That's the Online information environment Roy Soc report I entered higher up.

# Links from Helen
These may facilitate discussions this afternoon.

Our interventions policy provides more details on our role in making public interventions on the use of statistics. https://osr.statisticsauthority.gov.uk/policies/our-interventions-policy/

In October 2020, we started publishing quarterly management information related to our Casework. https://osr.statisticsauthority.gov.uk/casework/uksa-osr-casework-quarterly-management-information/

We also produce an annual summary of our casework which provides details of the volume and types of cases we have looked at. https://osr.statisticsauthority.gov.uk/publication/annual-review-of-uk-statistics-authority-casework-2020-21/

<span style="color: red"> Maths looks like this </span>

$$x^3 + y^3 = z^4$$

Lists look like this:

* **Item one**
* *Item two*
* Item three

Figures look like this:


# Bayesian model comparison

A concept that could be useful. Instead of black/white answer it produces a likelihood ratio. This is a first message, that perhaps we shouldn't be aiming to distinguish information into two categories but to use a graduated scale.

An example is the nice work to decide if the bones found under a carpark in Leicester were Richard III or not. The result of incorpating various strands of evidence was that the likelihood ratio was 6.7 million in favour of being Richard III. That seems to be considered as beyond reasonable doubt, though I don't know if law courts have a quantification (unlike the IPCC which has a conversion table between various phrases and probabilities). Furthermore, at least the English court of appeal has forbidden use of Bayes theorem except in connection with DNA evidence, which I consider a terribly backward step, analogous to the Catholic church condemning Galileo. How can it be right to forbid thinking logically with probabilities? Especially when the court's decision are formulated in terms of probabilities (civil courts use the balance of probabilities, meaning 50:50).

# Graduated scale for legitimacy
Adopt a probabilistic view and compute a probability p in [0,1] that the statement is true, or if preferred the odds ratio p/(1-p) [Warning: betting shops quote the odds against, (1-p)/p] or its logarithm (to whichever base you like). In civil courts the decision is in favour of the position with p>1/2 (balance of probabilities). In criminal courts conviction requires guilt beyond reasonable doubt, e.g. p>99.9%, though I don't think they ever specify a threshold.
To compute the probability is not straightforward, unless a mathematically specified problem. In general, one needs to start from a prior probability (which could be 1/2, but can allow a range of options and see how it affects the outcome). Then compute the posterior probability by multiplying it by conditional probabilities, but independence assumptions come in here.
A nice example is the bones in the carpark in Leicester: are they Richard III (or not)? DOI: 10.1038/ncomms6631 combines various evidence and comes to a posterior odds ratio of 6.7 million in favour.
Not sure how easy it would be to transfer this to assessing the legitimacy of statements the OSR is asked to address, but would be the direction I'd try.