# SMART Project Proposal

9.11.2020

Jakub Binkowski
Denis Janiak
Albert Sawczyn

SMART

<p align="center">
Full Project Proposal: <b> SMART Fact checking </b>
</p>

**Background**

- Tons of fake news and misinformations.
- Manual fact checking costs a lot and is time-consuming - there is no automation of this process in polish, even in a part.
- Distinguishing facts from opinions is not solved for polish.
- Searching in many sources for posts and articles takes a long time (fact checkers especially).
- Manual analysis of people’s stance towards a topic or a social issue is difficult.

**Scientific/technological questions of your project**

1. How effective is automatic fact checking based on similarity to Google Search headlines?
2. How effective is automatic fact checking involving combination of neural networks and knowledge base?
3. Can the polish Wikipedia be sufficient to build a knowledge base?
4. Which techniques allow us to build a high quality dataset for learning the multi-source recommendation system?
5. Are there noticeable differences in stance between verified facts and fake news?
6. How big is the problem of fake news in polish social media?

**Objectives of the project** (according to [[1]](http://examples.yourdictionary.com/examples-of-measurable-goals-and-objectives.html
))
- Build datasets for:
- automated fact-checking:
- fake/not fake classification (sourced from demagog.org.pl and wikipedia)
- knowledge source recommendation (sourced from demagog.org.pl and wikipedia)
- fact-check worthiness (wikipedia)
- stance detection (manually annotated)
- Build models designed for:
- supporting and automating fact checking
- performing social media opinion analysis through stance detection
- Create a common pipeline for solving all this tasks with reusable components (scrapers, preprocessing, models, features)
- Provide multivariate analysis and models for our collaborators

**The final product of the project**

The final product will consist of the proof of concept algorithm for automated fact checking of social media texts. We are going to provide datasets as well as analysis of social media based on the developed models.

**Scope**


**Data**

1. Demagog data
* politican's utterances containing factual claims
* data needs to be scraped from: https://demagog.org.pl/wypowiedzi/
2. Google Search API data
* identifying stance towards claims can help predict their veracity
* we want to associate each claim with supporting and opposing pieces of textual evidence
* transform each claim into sub-queries
* use these sub-queries with the claim itself as input to the search API
* calculate two similarity measures between the links’ content (documents) and the claims
3. Wikipedia data
* current Polish Wikipedia articles

**Timeframe** [[2]](https://www.projectinsight.net/project-management-basics/project-management-schedule).

| | Task | Start and End Dates |
| ---- | ---- | ---- |
| Phase One | Planning | 16/11/2020 |
| Phase One | Data acquisition | 30/11/2020 |
| Phase Two | Model implementation | 21/12/2020 |
| Phase Three | Solution tuning & validation | 18/01/2021 |
| Phase Three | Solution presentation | 25/01/2021 |

**Key Stakeholders**

| | Name |
| --- | --- |
| Consultation | NLP specialists (supervisors) |
| Consultation | professional fact-checkers in demagog |
| End Client | professional fact-checkers in demagog |
| Possible sponsor | Demagog Association |
| Project manager | Albert Sawczyn |

**Monitoring and Evaluation of Project Progress**

* Planning
- [ ] System architecture diagram - big picture of the system, main components, interaction between them
- [ ] Designed pipeline - choice of specific model architectures motivated by the current state of knowledge and data availability
- [ ] Evaluated articles - examination of results obtained by the authors, usability of techniques, availability of the open source code
* Data acquisition
- [ ] Implementation of a crawler for automatic Demagog's website scrapping (`dvc` storage, dataset versioning)
- [ ] Insightful analysis of demagog's data
- [ ] System which take a claim as an input and returns a stance/cosine similarity between the claim embedding and the embedding of google search results. Each fact should be linked with at least few articles.
- [ ] Acquisition of wikipedia references
- [ ] Implementation of a crowler of headers from Google search
* Model implementation - implementation of the entire pipeline for training and researching following models:
- [ ] Similarity & stance model
- [ ] Sources recommentation model
- [ ] Fake news classifier
* Model validation & tuning
- [ ] Use classification metrics to investigate model's performance
- [ ] Model should classify better than a random classifier
- [ ] Finetune model's hyperparameters
- [ ] Reiterate over model's implementation to improve model's performance if needed
- [ ] Qualititative analysis of models performance

* Solution presentation
- [ ] Poster presenting analysis, developed approach and obtained results

**The scientific method for evaluation of how good is your solution/analysis/service**

1. Overall methodology
- code versioned via `git` with well defined guidelines
- sophisticated CI which must fully pass
- datasets are clearly versioned and stored inside `dvc` - anlysis and models refer to specific version
- data used for training is analyzed first
- set of experiments should be performed in the following steps:
- well-defined hypothesis guiding the experiments
- experiments realization with outcome described below
- results summary
- chosen model easly accessible by the others, e.g. via `dvc`
- each trained model, has following expected outcomes:
- dataset used
- hyperparameters
- training logs: loss, metrics
- final evaluation of metrics
- checkpoints
- model training is done via Pytorch Lightning framework
- Jupyter Notebook (leveraged for sharing of documents containing live code, equations, and visualizations)

2. Classification metrics (fake news classifier / stance detection):
- precision
- recall
- F1
- accuracy
- confusion matrix

3. Sources recommentation system's metrics:
- Multilabel classification:
- Exact Match Ratio
- accuracy
- precision
- recall
- f1
- Hamming Loss
- Learning to rank
- Mean average precision (MAP)
- DCG and NDCG

**References**

Data sources:
- demagog.org.pl
- wikipedia.pl
- [Google Search API](https://developers.google.com/custom-search/v1/overview)

Main open source resources:
- standard data science tools: numpy, pandas, matplotlib, ...
- [spacy](https://spacy.io/)
- [huggingface-transformers](http://huggingface.co/)
- [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning)

[1] Khan, J. Y., Khondaker, M. T. I., Iqbal, A., & Afroz, S. (2019). **A Benchmark Study on Machine Learning Methods for Fake News Detection.** https://www.bbc.co.uk/news/resources/idt-sh/nigeria_fake_news

[2] Wang, W. Y. (2017). **“Liar, liar pants on fire”: A new benchmark dataset for fake news detection.** ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
(Long Papers), 2, 422–426. https://doi.org/10.18653/v1/P17-2067

[3] Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018). **FEVER: A large-scale dataset for fact extraction and verification.** NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1, 809–819. http://arxiv.org/abs/1803.05355

[4] Augenstein, I., Lioma, C., Wang, D., Lima, L. C., Hansen, C., Hansen, C., & Simonsen, J. G. (2020). **MultIFC: A real-world multi-domain dataset for evidence-based fact checking of claims.** EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, 4685–4697. https://doi.org/10.18653/v1/D19-1475

[5] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). **Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.** http://arxiv.org/abs/2005.11401

[6] Sorower, M., & Sorower, M. S. (n.d.). **A Literature Survey on Algorithms for Multi-label Learning.** Retrieved November 8, 2020, from https://www.researchgate.net/publication/266888594

[7] W. Antoun, F. Baly, R. Achour, A. Hussein and H. Hajj, **State of the Art Models for Fake News Detection Tasks**, 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), Doha, Qatar, 2020, pp. 519-524. https://ieeexplore.ieee.org/document/9089487