---
# System prepended metadata

title: Benchmarking
tags: [benchmarking]

---

# Meeting 13 Dec 2023

## To discuss
1. Scope of the pipeline + general workflow
    Germline variants
    Somatic variants
    Structural variants
    CNVs
    Single cell?
    MSA?
    atacseq?
    all outputs of pipelines??
3. Proposal?
4. What is the overlap with [NCbench](https://f1000research.com/articles/12-1125) and how will we handle this?
    (NCbench currently benchmarks germline small variants.)

## Notes
1. NCbench
    2. snakemake pipeline dependent on the type of analysis (evaluation), includes the production of the matrices used for the benchmarking. (https://github.com/snakemake-workflows/dna-seq-benchmark)
    3. Github actions independent of analysis (https://github.com/ncbench/ncbench-workflow/blob/main/.github/workflows/main.yml)
    4. Reporting (https://github.com/ncbench/ncbench.github.io/blob/main/.github/workflows/static.yml)

- Use NCbench for germline and somatic variants
- Add pipelines (nextflow or snakemake) to the NCbench organization for other data types
- Possible benchmarking pipelines by nextflow: RNA-seq variants, single cell  variants, SV - CNV variants
- Instead of one major benchmarking nf-core pipeline, it will be easier to keep seperate nextflow pipelines by data type. 
- We could try with one or a couple of PoC and see how it works.

# Benchmarking

## Variant calling

### SNVs
- `hap.py` (https://github.com/Illumina/hap.py)
- `RTGtools vcfeval` (https://github.com/RealTimeGenomics/rtg-tools)
- `bedtools jaccard` (https://bedtools.readthedocs.io/en/latest/content/tools/jaccard.html)

GH4GH Standards for small variant benchmarking(https://github.com/ga4gh/benchmarking-tools/tree/master/doc/ref-impl)

### SVs 
- `truvari` (https://github.com/ACEnglish/truvari)
- `SVanalyzer` (https://github.com/nhansen/SVanalyzer/blob/master/docs/svbenchmark.rst)

### Somatic Variants
- `som.py`(https://github.com/Illumina/hap.py/blob/master/doc/sompy.md)


### Normalization tools
