Technicalities

• The course is taught at the faculty (i.e. is stationary), with Zoom kept as a backup plan
• Course materials will be available via Google Drive / GitHub / on lecturer's website
• Installing Git and registering to GitHub (or a similar service) will be necessary later on
• Hit us up in case of trouble
• If you fall behind (especially during first few classes) you might have problems with catching up

A roadmap of sorts (key elements of the course)

• What's the point anyway?
• Version Control Systems (Git - used throughout the course)
• Quarto and Markdown
• Ground rules (clean code, documentation, etc.)
• Tools (i.a. Kaggle, Repositories)
• Reproducible Environments, etc.
• Testing
• Metaanalyses
• Your projects

The Three Rs (Peng, 2011)

Repetition

Check if the results hold when the 'experiment' or process is exactly repeated

Reproduction

E.g. new researchers reach the same outcome (with same data)

Replication

Check if results hold in a different setting / with different methods / frameworks, etc.

The Five Rs? (Benureau and Rougier, 2018)

The code should be

• Re-runnable (executable)
• Repeatable (producing the same result more than once)
• Reproducible (allowing an investigator to reobtain the published results)
• Reusable (easy to use, understand and modify)
• Replicable (acting as an available reference for any ambiguity in the algorithmic descriptions of the article).

Peng (2011) as a graph (note: it's a spectrum)

Source: Camfield & Palmer-Jones (2013) - Three "Rs" of Econometrics: Repetition, Reproduction and Replication

Why replicate studies?

1. Rob, R. and Waldfogel, J. (2007). Piracy on the Silver Screen. Journal of Industrial Economics.

2. Bai, J. and Waldfogel, J. (2012). Movie piracy and sales displacement in two samples of Chinese consumers. Information Economics and Policy.

3. Herz, B. and Kiljański, K. (2018). Movie piracy and displaced sales in Europe: Evidence from six countries. Information Economics and Policy.

4. Ende et al. (2018). Global Online Piracy Study. Google Report.

Relationship between piracy and sales across studies

Even more similar studies

Why reproduce research / make research reproducible?

• Transfer of knowledge
• Replication
• Work
• Own workflow
• Check for errors
• Internal consistency
• etc. (all good reasons)

Why is research often (regrettably) not reproduced?

• Reproducing research regarded as 'unoriginal'
• Time might be considered 'wasted'
• A succesful reproduction does not contribute to a researcher's 'portfolio' (no returns to the work input)
• Might create tensions with other team members / requires a workflow
• Invalidate priors to other conducted work
• Researchers have little or no incentive to publish code and data to make research reproducible (fortunately it is changing in the right direction)

see: Camfield and Palmer Jones (2013).

However, reproduction is important

See "Replication Crisis":

A 2016 study in the journal Science found that one-third of 18 experimental studies from two top-tier economics journals (AER and QJE) failed to successfully replicate.

Also: you can browse through some non-replicable (and sometimes retracted) marketing studies here, but note that it's not a random sample.

So what's wrong with research? (1)

Publication bias or the file drawer problem:

• tendency to submit/publish statistically significant results, in-line with prior research
• tendency to omit insignificant results and unconfirmed hypotheses
• less intuitive results might take more time to publish (we don't see them yet)

As explained by XKCD.

So what's wrong with research? (2)

p-hacking and questionable research practices:

• fiddling with model specifications (models, final sample, choice of control variables)
• leaving out contradicting results
• writing down the hypotheses after the results are in (HARKing)
• creative interpretation of p-values

Quick guide by XKCD and even a related R package.

So what's wrong with research? (3)

Even if there were no biases… Let's recall statistical testing.

We need a null hypothesis, e.g. that a coefficient is equal to 0.

And an alternative, e.g. that it's larger than 0.

If the null is true, we'd expect our test statistic to be somewhere around the 0 point.

If it's very far away, we conclude that it's probably from a different distribution after all.

What's wrong with research? (3)

That's testing in a nutshell.

We typically choose a cutoff (e.g. 5%) for the test statistic and compare it with the p-value.

Two ways we can go wrong though:

• Type I error: reject null although it's true (rare things can happen)
• Type II error: not reject null while it's false

And so by statistics alone, we err every now and then.

Take a look at the XKCD strip again and count the 'testing' frames. That's a case of a Type I error followed up with selection bias. Here's a study on that as well.

Also other stuff, like

Bad coding practices

First, we retrieve and analyze more than 2000 replication datasets with over 9000 unique R files published from 2010 to 2020 [at Harvard Dataverse]. (…) We find that 74% of R files failed to complete without error in the initial execution, while 56% failed when code cleaning was applied, showing that many errors can be prevented with good coding practices.

Trisovic et al. (2022)

Solutions - systemic

Registered studies (mainly in medical journals, increasingly in other too)

Study marking (e.g. Biostatistics journal and its "C", "D", and "R")

Data / code sharing

More strict p-value requirements

Publishing replication studies

Solutions - as authors

Meticulous code with comments, made publically available

Well-documented process, environment, etc.

We'll cover some ground rules later in the course

Here's a call to ease up with the interpretation of p-values (and it's not the first or last): Scientists rise up against statistical significance

(why proper descriptions are so important…)

The influence of hidden researcher decisions in applied microeconomics

(…) Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well.

Solutions - meta-research or a study of studies

• Aggregates numerous existing studies and analyses them jointly.

• Different goals:

  • find the true effect size,
  • verify biases in the literature,
  • reduce Standard Errors,
  • identify new relationships.

• Requires detailed and meticulous documentation of the (often arbitrary) procedures -> for reproducibility.

• There are also metaanalyses of metaanalyses… (one last XKCD - for today). Here's somethin of this sort: https://www.bmj.com/content/344/bmj.d7762

Meta-research

Basic approaches:

• find a common measure across studies (coefficient in same units, a t-statistic or p-value, a difference etc.)

• regress it on study/data/author/period/etc. characteristics

• look at distributions of statistics

• search for irregularities or inconsistencies

• funnel plots

Meta-research - example

Krawczyk, M. (2015) - The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature. PLOS ONE 10(6).

Studied >135,000 p-values from >5,000 studies.

Found that "some authors choose the mode of reporting their results in an arbitrary way"

• "they frequently report p values “just above” significance thresholds directly, whereas other values are reported by means of inequalities (e.g. “p<.1”)"
• "they round the p values down more eagerly than up"
• "[they] appear to choose between the significance thresholds and between one- and two-sided tests only after seeing the data"
• "about 9.2% of reported p values are inconsistent with their underlying statistics (e.g. F or t)"
• "it appears that there are “too many” “just significant” values."

"Fig 2. Distribution of directly reported p values (restricted to .001–.15, equal weight of each paper)." from Krawczyk (2015)

"Fig 3. Distribution of re-calculated actual p values (restricted to .001–.15)." from Krawczyk (2015)

Meta-research: example 2

"Effect sizes reported in the biomedical literature" from Monsarrat and Vergnes, 2018

Passing the course: Final project (40 pts.) + Presentation (10 pts.)

• In teams of 2, with teamwork graded
• The topic choice has to be confirmed with the coordinator
• Topics at the end of this presentation and in course materials
• Must-haves for all projects:
  1. project stored in a repository from the start (viewable history of well-described contributions from all team members who collaborate) (15 pp.)
  2. code and results with appropriate documentation (e.g. Markdown) appropriate for full reproducibility (including software version, etc.) (15 pp.)
  3. code in a clean and user-friendly format (10 pp.)

The core idea of the project

The project isn't aimed at testing your programming, econometric or statistical skills. The correctness of your analytic reasoning will not be graded in this course.

Instead, your projects should focus on:

1. the process of reproduction (did it succeed? what where the challenges? what was missing in the original source? what didn't work? what could have been done in the original work instead?)
2. making your own process reproducible
3. documentation and good coding practices
4. collaboration and version control via Git

More about the rules: plagiarism

No tolerance for plagiarism, including self-plagiarism (!)

Plagiarism is not only about providing a source, it is also about copying large parts from one project to another, especially without adequate modifications.

Project topics (and scope!) are agreed upon via e-mail. If you change them without approval, your project will not be graded (or will be graded accordingly with the initial scope that was agreed upon).

More about the rules: AI policy

This course is not about your programming skills. It's about understanding the importance of reproducibility and the ways of achieving it.

You can use AI (e.g. LLMs) to support your writing and coding. If so, please state:

• the scope of the support,
• the AI model and version.

(Also cite other online sources, even if only of code snippets)

More about the rules: teamwork

Work in teams of 2. If you can't find a team by the time of the deadline, you will be assigned to a team.

If you run into any problems regarding your collaboration, address them on the go. You may involve the course coordinator if so.

Do not wait until the deadline to notify the coordinator about not having been able to work with other team members.

Passing the course: assignments (50 pts.)

Along the way, we'll be (sometimes) doing class assignments (i.e. during classes).

• You'll be asked to upload them to an online repository at the end of the class (as a prerequisite for having it counted).
• Not every week
• It's not about finishing everything, but about checking activity during the classes
• Each activity will be counted as done (1) or not done (0). At the end, the total will be rescaled for a maximum of 50 points towards the final grade.
  • i.e. if you do 4 out of 7 activities, you'll get 4*(50/7) = 28,6 , i.e. 29 pts. towards the final grade.

Passing the course: Presentation (10 pts.)

• Last two meetings of the course will be devoted to the presentations.
• The presentations should last app. 20 minutes, include all team members and briefly go over:
  • what the project is about, how it was done, how it works, etc.
  • include a short presentation of the code as well.
• The projects will be run from a computer in the lab, after pulling them from a GitHub repository.

Deadlines

• March 29, 2024 -> send information about teams (send an e-mail, including names of the members)
• April 19, 2024 -> send a link to the team's GitHub repository (or invite via GitHub)
• May 6, 2024 -> confirm project topics (send an e-mail with the topic for confirmation).
• June 14, 2024 -> no further changes in the project repository allowed.

You can finish the project earlier and ask for an earlier grading as well.

Thank you!

Topic Examples

1. Take an econometric / statistical analysis (e.g. from your bachelor thesis or Kaggle):

• Translate the codes to a different programming language (e.g. R to Python or Stata to R, etc.)*
• Reproduce the results.
• Pick a way to improve the study or update the data** or perform a robustness check and do it.
• Discuss the findings***, potential problems, inconsistencies and conclusions.

*you should try exploiting the advantages and functions of the 'new' language
**data update can include: collecting new survey data (not many), using newer datasets, etc.

*** you should provide a discussion on whether you managed to reproduce original results accurately or not and what might be the reasons. Also discuss whether your extension changed the conclusions and why that might be.

2. Take a published (in a peer-reviewed journal, not Kaggle notebook) research paper with no code attached. Reproduce its (main) findings. Report on the problems along the way.

3. Take a simple meta-analysis study (examples below). Then do the following:

• Reproduce the obtained results using the reported sample of studies.
• Add 2-5 newer studies, preferably using the selection process reported in the original study.
• Replicate the results with the extended sample.
• Describe your findings and discuss them.