The Bayesian Conspiracy - Academic Publishing

Recorded 2023-05-14

My infrequently updated blog: https://blog.pan-narrans.xyz featuring: Why multi-panel figures are terrible & we should stop using them. & Showing our working
Contact: richardjacton@pan-narrans.xyz (see: https://blog.pan-narrans.xyz/about.html for alternative contact info)

Ageing & Immortality Special episode: https://www.xenothesis.com/53_xenogenesis_ageing_and_immortality_special/

Data: Inception to Publication & Beyond an ebook I recently wrote for work using some of the tools and practices discussed in the episode on the research data lifecycle relevant to anyone wanting to get started with this way of working.

  • last time we spoke was after I'd submitted my thesis, but before I graduated with my PhD shortly after I'd moved to Germany
  • My opinions are my own and necessarily reflective of those of my employer
  • Joined up with the guild of the ROSE during the pandemic, now been sporadically doing some guild podcast episodes with David Youssef 'oasis of rest', some interviews with guild council members some of just me and David chatting. - more coming soon
  • To preface what will be a lot of critiques: My overall outlook on the future of academic publishing is generally positive.
    • Lots of people chaffing in the current system seeing quite a lot of new ideas and entrants into the field of alternative models, however I've seen limited success so far.
    • I'm frustrated by the glacial pace of change.

Academic publishing has problems at every level of analysis from the economics to article formatting, from the abstract to the mundane.

  • Background on the academic publishing market: The Economics of Academic Publishing

    • it's an oligopoly - like all the others (5 big companies, Springer nature, Taylor & Francis, Wiley, Sage, Elsevier - 40% profit margins)
      • Monolopy & Monopsony (Chokepoint - Rebecca Giblin and Cory Doctorow)
        • Scientists as creative workers - they are artists constrained by the medium of empiricism, but they nevertheless still often possessed of an artistic temperament. "vocational awe"
      • But academics are both the suppliers and the consumers, oh and also most of the labour
        • Academics are exploited labour - in a better position that most because they are subsidised by the state via universities, and having rent extracted as both suppliers and consumers - ironically we basically did it to ourselves, collective action problem / inadequate equilibrium / multi-polar trap.
          • Everyone is convinced everyone else cares about publishing 'highly' so every one still tries to.
          • losses passed on to the public purse at the taxpayers expense + some philanthropists and businesses that fund research.
      • To be clear we are producing the content, doing most of the work moderating, critiquing, and evaluating that content, and the forking over our copyright to that content to the journals who then sell it back to us.
    • Value add of journals?
      • Hosting the papers (inexpensive, largely redundant with public archives like pubmed central, when full texts are available)
      • Copy editing (if you are lucky, and substantially automatable)
      • Admin of review and other aspects of the publication process, (now mostly automatable)
      • Some curation, but with unhelpful systematic biases (file drawer effect, other forms of publication bias)

    • Business models

      • Classic: subscriptions - closed
      • New: pay-to-publish - open-access
        • On the plus side people outside academia can read more papers now
          • Also matters for access to institutions in poorer countries who couldn't afford to READ the journals
          • sci-hub pirated a bunch of papers which did improve read access to people who couldn't pay
            • Showed up the publishers badly because the site was a better user experience despite the founder (Alexandra Elbakyan) being on the run and in hiding from the international copyright cartel
        • APCs (article processing / publishing charges)
          • $10,100 to publish an article in Cell (as of 2023-04-14) https://www.elsevier.com/about/policies/pricing
            • Median $2990 from 2619 journals for which fees were provided, minimum $200 (before tax)
          • What do they do for this money?
            • Run a shitty website
            • Send a few emails
            • Provide the worlds worst value for money copy editing service that it's even good
          • Many 'predatory?' 'more predatory' journals which will publish your papers for a fee but you won't get any clout as a result

    • F*ckery

      • Free labour from authors & reviewers - sign over copyright
      • Bundled subscriptions, just like cable you can't only buy the good channels you have to pay for a bunch of crap also
        • With the open-access 'victory' the big publishers just pivoted to doing institutional subscriptions to publication fee bundles instead of subscription fee bundles.
      • NDAs on price negotiations with university libraries

  • Peer-review

    • I agree with Adam Mastroianni about most of this but I have a minor point of disagreement about 'science is a strong link problem' - pre-publication review can die a quick and unceremonious death and I will not mourn
      • strong-link/week-link is too black and white a framing "There are two kinds of people in the world. Those who think there are two kinds of people in the world, and those who do not"
        • In the long term and in retrospect science is a strong link problem
        • paradigm shifting science is a strong link problem
        • ordinary science (Kuhn) is less so
          • important to be able to have some degree of trust in information in the literature around periphery of your core questions - this may be related to differences in out fields.
        • "pitt of sucess", not a "pinacle"" we can't expect a level of quality it takes a heroic effort to achieve in quantity.
          • Defaults are king - if processes that are costly signals of quality are normalised we can bring up baseline trust in the literature
          • Importantly we can make that trust more granular as we can more accurately asses specific assertions if we have more open processes & smaller units
    • As with all media no longer about what gets published so much as about what gets boosted by being in the prestigious journals. There is a scale and content curation problem that didn't exist when it was a few dozen guy's writing letters to one-another.

  • Is the reproducibility crisis also a thing in biology - YES its not just the social sciences

    • [[Reproducibility Project: Cancer Biology]]
    • [[@Rigor mortis: how sloppy science creates worthless cures, crushes hope, and wastes billions]]
    • [[@Science fictions: exposing fraud, bias, negligence and hype in science]]
    • Ben Goldacre: Bad Science 2016, Bad Pharma 2012

Fixing things

  • Building parallel systems, adopt these alongside conventional publishing, make it irrelevant
    • pre-prints
      • But better if you can, this is time consuming - write up your work in a format that is actually readable, fix figures and writing style.
    • pre-publication is Unacceptably slow The Speed of Science Saloni Dattani
  • Move to post-publication review but not merely this: review should be open, interactive, & ongoing
    • Open Science Movement
      • open FAIR data, open notebooks, open code, open protocols, open reviews etc.
      • Open Science Community remains somewhat disjoint.
        • Quite a few Orgs doing their own thing
        • A promising community with some good information exchange I recently found is the The Turing Way who are collectively authoring a book on open science.
    • Open Review
      • "with enough eyes all bugs are shallow"
      • Transparent process, we see the back and forth, the changes made and the reasons why
        • like the Wikipedia discussion & history - it's very illuminating about the points of contention.
      • Standard counter:
        • People will be afraid to criticise more senior academics in public especially if they might have an impact on their career progress, anonymity protects them.
        • in practice, people can often work out who their reviewers are
      • Interesting experiment here F1000 now owned by on of the giants Taylor and Francis
      • peerj
    • interactive review
      • It's 2023 why are we writing letters to one another via an intermediary to review a document?
      • Take a leaf out of open-source software development's book and use something like git and git collaboration platforms like gitlab.
      • code review like approach - going back and forth on issues and merge requests
        • The tooling sucks, some efforts to improve / open this up but mostly re-capitulates existing poor workflows
          • eLife: good intentions but they've been a bit all over the place in my view, came in a bit too ambitious with their software plans, pivoted to a different publication workflow and shifted technical focus 'Libero' & stencila
          • janeway
          • Open Journal Systems
          • target learned & scholarly societies who publish journals 'the proceeding of x'
            • adoption of new workflows and practices, aim to reduce workload & cost for them and increase their control compared with partnering with the big publishers. potential to inovate
            • open federated platform
    • Attempts and reform and alternative approaches
    • Review Bounties - (my original contribution, as far as I've been able to ascertain)
      • inspired by 'bug bounties' and 'read teaming'
      • instead of an APC issue a review bounty
        • 'journal editor' is replaced by a review arbiter and matchmaker
          • They tie break disagreements between reviews and authors when they occur and aid in finding suitable reviewer, may also provide hosting for the article and compute to validate computational results.
        • Agreed cut goes to, review arbiter, reviewers and remains as a bug bounty which can be claimed if a bug that materially alters the conclusion(s) of the work is discovered (according to a pre-agreed decision process)
          • Accrues back to the author and reviewers over time, this rewards good work and incentivises rapid bug discovery.
        • Funders currently have no simple way to incentivise scrutiny on published works and ensure quality - minimum bug bounty percentages/amounts could be mandated
        • 3rd parties with an interest in the work could add to the bounty pool if they want to make an investment based on a conclusion in a work.
      • ties into my suggestion of the new business models for academic publishing
        • open-source decentralised publishing platforms
          • Open source is essential to avoid the trap of bad software stewardship from a monopoly platform. - if you deliberately have not 'moat' you can't be a dick or someone will kick you out of the castle. accountability mechanism. favours cooperation over competition
        • federated network
        • Nextcloud is perhaps the best example of the business model I'd like to see replace the current one (Redhat, SUZE, and much smaller ventures like Taiga & eLabFTW)
    • semantic versioning for papers. Versioned releases 'pre-print' 0.0 series, peer-reviewed 1.0 series, new data major version bump, errata and corrigenda that affect the substantive conclusions minor version bump, inconsequential typos etc. minor version bump
    • Attempt to quantify the 'value add' of peer-review - it's small Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature
      • It could be better
        • Reviewers are prone to asking the wrong questions and offering poor suggestions for actually improving the quality of the work - not always you sometimes get really good helpful and thoughtful responses.
          • Good review is high effort and there is little reason to do it well
          • "You haven't cited this other thing I think is relevant (that oh BTW I wrote)"
            • sometime this is relevant but it's hardly the more important critique
          • "Does this generalise - you've shown it in worms but how about mice" - valid question but that's a whole different paper a a lot of work.
          • questions I'd like people to ask but will probably not go over well ATM
            • "What is the protocol for this method in this experiment?"
            • "Could I get a copy of the code you used in this analysis, follow up I had some trouble running it could you document this as part of your "
  • pre-registration / registered reports
    • file drawer effect / publication biases

Units of publication

  • Review-ability

  • Hierarchy of publication types

    • Traditionally:
      • Primary papers
      • Reviews
      • Textbooks
    • What I think this could/will look like
      • pre-registrations / experimental plans/designs
      • Data, Methods, Protocols, Pipelines, & Software publications
      • Experimental results
      • Theory, synthesis and prediction (without a requirement for experimental detail but ideally with resolution criteria)
      • Topic Reviews, Curated Best practice resources for lab protocols and computational pipelines as well as, Benchmarking resources
      • git-books / wiki-books - much like a textbook but under continuous revision
      • Science communication (multimedia, curricula, blogs, policy briefs etc.)
    • Alternative publications types / platforms

  • nano-publications, linked data, and machine interpretability

  • Formatting

    • separating the semantic content from it's formatting (HTML & CSS)
    • word limits on methods, limits on the number of citations - connected to paper length

  • Figures

  • Reproducibility / Replicability

    • publishing data and protocols
      • bad incentives
    • Experimental
      • Protocols sharing and collaboration platforms e.g. protocols.io
        • Could tie in to publishing of informal replications - it is common to replicate other's experiments informally when setting up your own version their experimental system to extend it for your own experiments.
      • JoVE (journal of visualised experiments) - and less formal approaches
    • Computational
      • Data
      • Code
      • Computational Environment
    • Computational Reproducibility Tools
    • publication and review workflow

  • Publication metrics

    • Alternatives to 'impact factor'
      • impact factor = number of citations / number of articles published (in a given time period, often 5yr rolling)
    • altmetrics - of questionable alignment
    • plaudit
    • ~ kind of pubpeer

  • Grants

    • Conventional Grants
    • prizes
    • salaried positions
    • 'Venture research' - Don Braben

  • Credit and Authorship

    • rescognito "Rescognito is a free service for recognizing and promoting good research citizenship. With Rescognito you can acknowledge colleagues (and be acknowledged by colleagues) for meaningful contributions to scholarly research."
    • CRediT (Contributor Roles Taxonomy) "is a high-level taxonomy, including
      14 roles, that can be used to represent the roles typically played by
      contributors to research outputs. The roles describe each contributor’s
      specific contribution to the scholarly output.""

Further listening

Select a repo