# The brief
* [Original paper](https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001421)
* [Suppl data from original paper](https://figshare.com/articles/dataset/Additional_material_for_article_Exploring_bacterial_diversity_via_a_curated_and_searchable_snapshot_of_archived_DNA_sequences_/16437939)
* [mof paper](https://www.biorxiv.org/content/10.1101/2023.04.15.536996v1)

Some time back "we" assembled 661k bacterial genomes (everything in the ENA/SRA up to end 2018) and curated/QC-ed it and so on, and made sourmash/ppsketch/COBS indexes available. 

This was great (I am biassed) but time moves on so we've decided to update the dataset. We're (Leandro Lima is) assembling all bacteria (isolates not metagenomes) in the ENA up to ~now, which will top out around 2 million, depending on how much we chuck out for QC reasons. The vast majority of the assemblies will be done in about a week

Recent work from Karel Brinda and Mike Baym (and our team)  means we can compress these genomes much better than before (Grace's data, zipped, is normally  ~900Gb, but this can be shrunk losslessly to ~20Gb), and Karel's mof-search will actually do full alignment to that dataset even on a laptop.

This time round I'd like to:

a) make sure we succeed in having a web-interface allowing people to search
b) invite wider collaboration. We'll run centrally anything that's the same across all the genomes (assembly, QC, contamination check, annotation, pan-genomes), but I'm hoping to find others will join us:

Eg 1 - species experts who want species-specific stuff run (eg kleborate on klebs), and who might join us  in running/curating that stuff. I'd like to ensure everything is reproducible, but we can work together on that. We have only haphazardly started contacting people so far (Kat Holt abou klebs, Kate Baker about shigella, Nick Thomson on enterics generally), and would welcome contact from interested people. I will paste below the number of genomes of different species we're assembling (modulo QC, and the fact that ENA metadata is full of noise/errors).

Eg 2 - mobile element and defence against the dark arts experts. Would be great to run tools like DefenceFinder and talk to people who have time/energy to look through the output of these and help curate/deal with issues.

Eg 3 - prophage annotation and CRISPR spacer annotation.
Current plan is (ill-thought-through so subject to change) - to make the assemblies available via zenodo (or maybe EBI ftp) once QC is done, but before all the rest is done. I expect the pan-genome construction will be tricky at this scale (panaroo would not work at this level, so am v happy Gerry Tonkin-Hill and John Lees are involved), and I don't want to delay availability until everything is done. Thanks to Karel's work, we ought to be able to compress 2 million genomes into 100Gb, and provide tools so people can align to the full dataset locally on a laptop if they want, so they dont need to wait for the web version. 

So the potential scheme is:

1. assemble and QC the genomes and make mof index. We will submit the assemblies to the ENA but accessioning is slow, and I don't want to wait for that
2. stick it all on zenodo so people can use them.
3. then spend months/whatever doing the annotation
4. species-experts can help us run some  species-specific tools collecting more data.
5. somewhere in here, make the effectively-BLAST work via the web
6. then finally write a data paper one day
Please contact me (here or via email) if you're interested in joining in!

# Wish list
Add more ideas to the wishlist, and eventually we can assign these ideas as objectives into the road map below.

## Species specific

* Stx typing (*E. coli*)
* Pathotyping (*E. coli*)
* Phylogroup (*E. coli*)
* Looking for new subspecies (*Salmonella*)
* Typing *Bacillus cereus* group
* MTB: TB annotation is wrong at something-something
* Serotyping on the pneumococci
* in silico serotyping (*E. coli*)
* in silico serotyping (*Salmonella*)
* SCC*mec* typing (*S. aureus*)

## Other

* Comparing `skeska` and `shovill`. i.e. this effort vs ncbi pathogens.
* Flagging accessions that reflect natural diversity, i.e. not lab mutagenesis experiments
* Flagging accessions that meet a minimum quality (not contaminated, QC pass)
* PLASMIDS
* replicon typing?
* mob typing?
* mpf typing?
* pMLST?
* ptu?
* lociq typing?
* Dereplicate the genomes? I bet a lot of those genomes are from outbreaks
* MLST all the things
* Integrating future sequence data
* Harmonised gene names
* Generic annotation
* Annotation with genus-specific databases
* `mof-search`
* AMR gene/SNP detection?
* CRISPR spacer annotation
* Prophage annotation
* Biosynthetic gene cluster (BGC) detection (e.g., with [GECCO](https://gecco.embl.de/))
* Source attribution
* Making assemblies easily accesible for workflow managers (simple download/extraction scripts for nextflow/snakemake/wdl)
* maybe of interest: find assemblies with unique or rare genes (or ORFs) (by some PID cutoff)
# Road map
Feel free to rearrange ideas between milestones. While the milestones imply a sequential order, there is no reason why a splinter group couldn't run with an idea in a later milestone, while the main group concentrates on more fundamental ground work.

## 0. Initial prototype
* Release of assemblies as Microbes on a Flash Drive index (compressed assemblies) on:
* Zenodo
* ENA FTP (should be 100Gb in size).
* Release of minimal contextual data table (as a delimited file)
* Release of suggested data standard for complete contextual data
* Guidance on how to contribute results - data provenance (tools, versions, should most things be containerised for reproducibility?) (zam says - i would like all processes that have been run in one repo, and containerised ideally yes. that way we can have computation done in multiple places, but have confidence in repro)
* Guidance for authorship and participation - consortium model like coguk or case by case invites?

## 1. Minimal viable product
It maybe wiser to concentrate on generic processes first.
* Distribution methods
* Zenodo
* Cloud storage (s3://all-the-genomes/ERR..fasta)
* Light weight API (with searchable metadata and links to static files - [FastAPI](https://fastapi.tiangolo.com/)?)
* [Open data sponsoship through AWS](https://aws.amazon.com/opendata/open-data-sponsorship-program/) *- this only lasts 2 years though?*
* CLIMB?
* Established roadmap and terminology (incl terminology for types of deduplicated dataset (eg absolutely all genomes, dedup-ed to remove almost identical, representative genomes from each pop-punk cluster))

## 2. A Full Version
* Release of genome annotations which ideally are harmonised within genus so gene names are consistent
* AMR gene detection
* Some MLSTs?
Species specific focus.
* Initial per species stuff.
* search/alignment

## 3. Second iteration
* more species-specific stuff

## 4. Stretch goals

# Potential tools of interest

* [PopPUNK](https://poppunk.readthedocs.io/en/latest/)
* [Assembly Dereplicator](https://github.com/rrwick/Assembly-Dereplicator)
* [Alphafold might be good also](https://github.com/iqbal-lab-org/AllTheBacteria/issues/7) but it has been pointed out to Zam that a *lot* of bacterial genes are already alphafolded.
* [Bactopia](https://bactopia.github.io/latest/)
* [assemble-all-ena](https://github.com/iqbal-lab-org/assemble-all-ena)

## Annotation
* [Prokka](https://github.com/tseemann/prokka)
* [Pharokka](https://github.com/gbouras13/pharokka) - Fast Phage Annotation Tool
* [Bakta](https://github.com/oschwengers/bakta)
* [ggCaller](https://github.com/samhorsfield96/ggCaller)

## Typing

* https://github.com/B-UMMI/seq_typing/tree/gbs_types
* https://github.com/BenJamesMetcalf/GBS_Scripts_Reference

| Genus/Species | Tools |
|---------------|-------|
| *Bacillus cereus* group | [BTyper3](https://github.com/lmc297/BTyper3) |
| *Escherichia* / *Shigella* | [ClermonTyping](https://github.com/A-BN/ClermonTyping)*, [ECTyper](/bactopia-tools/ectyper/), [EzClermont](https://github.com/nickp60/ezclermont), [ShigaTyper](/bactopia-tools/shigtyper/), [ShigEiFinder](/bactopia-tools/shigeityper/), [STECFinder](https://github.com/LanLab/STECFinder) |
| *Haemophilus* | [hicap](/bactopia-tools/hicap/), [HpsuisSero](/bactopia-tools/ssuissero/) |
| *Klebsiella* | [Kleborate](/bactopia-tools/kleborate/) |
| *Legionella* | [legsta](/bactopia-tools/legsta/) |
| *Listeria* | [LisSero](/bactopia-tools/lissero/) |
| *Mycobacterium* | [Mykrobe](https://github.com/Mykrobe-tools/mykrobe), [TBProfiler](/bactopia-tools/tbprofiler/) |
| *Neisseria* | [meningotype](/bactopia-tools/meningotype/), [ngmaster](/bactopia-tools/ngmaster/) |
| *Pseudomonas* | [pasty](/bactopia-tools/pasty/) |
| *Salmonella* | [SeqSero2](/bactopia-tools/seqsero2/), [SISTR](/bactopia-tools/sistr/), [Genotyphi](/bactopia-tools/genotyphi/) |
| *Staphylococcus* | [AgrVATE](/bactopia-tools/agrvate/), [spaTyper](/bactopia-tools/spatyper/), [staphopia-sccmec](/bactopia-tools/staphopiasccmec/) |
| *Streptococcus* | [emmtyper](/bactopia-tools/emmtyper/), [pbptyper](/bactopia-tools/pbptyper/), [PneumoCAT](https://github.com/ukhsa-collaboration/PneumoCaT), [PneumoKITy](https://github.com/CarmenSheppard/PneumoKITy), [SsuisSero](/bactopia-tools/ssuissero/) |

"*" not available from Bioconda

## Pangenome tools

* [PanTA](https://www.biorxiv.org/content/10.1101/2023.07.03.547471v1) is an efficient method for constructing and analyzing bacterial pangenomes. It provides a 10x speedup and uses 2x less memory compared to existing methods. PanTA allows progressive pangenome construction, reducing memory requirements. It can build the pangenome of Escherichia coli using a laptop within two days, showcasing its scalability.

* [Pandora](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-021-02473-1) is a method for allowing SNP/indel and gene presence calling based on a reference pan-genome graph.

# Contacts
Please mention who you are and what you would like to do.
* Zam (speaking also for Leandro Lima) - assembly, QC (including tax checks), building MOF/COBS search index, building pandora reference graphs for top-N species.
* John Lees (has volunteered, Zam putting his name here) - poppunk, gene annotation and harmonisation within species (genus?).
* Nabil-Fareed Alikhan; *E. coli*, *Salmonella*, Distribution, metadata, API development
* Natacha Couto; *S. aureus*, *Klebsiella*, *E. coli*, other Enterobacteriaceae, metadata, QC, AMR curation
* Robert Petit; packaging (conda-forge, Bioconda), workflow management (Nextflow), API, breaking things
* Laura Mišela Carroll; *B. cereus* group typing/taxonomic assignment/annotation; BGC detection
* Mike Beeton (volunteered by Natacha, with permission); Mycoplasma and Ureaplasma
* Julio Diaz Caballero (volunteered by Natacha, with permission); Pseudomonas aeruginosa, metadata, QC, assembly
* Michael Hall - reference graphs (in collab. with Zam). Will require working closely with John and Gerry re: annotation
* Matthew Croxen - QC, typing, CPO/CRE/CPE and other microbes of public health concern