---
# System prepended metadata

title: GeneLab metagenomics workflow example run
tags: [GeneLab]

---

---
tags: GeneLab
title: GeneLab metagenomics workflow example run
---

# GeneLab metagenomics workflow example run

---

> General workflow usage info can be found in the [NASA GeneLab repo here](https://github.com/nasa/GeneLab_Data_Processing/tree/master/Metagenomics/Illumina/Workflow_Documentation/SW_MGIllumina). **This page streamlines that a little and includes explicitly grabbing and running a small example dataset.** This workflow as currently written relies on conda and snakemake. 
> 
> The metagenomics workflow does rely on large reference databases. They will be installed and setup automatically by the workflow the first time it is run. All together, after downloaded and unpacked, they will take up about 240GB of storage, but they also may require up to 500GB during installation and initial un-packing. 
> 
> Due to the large time required for downloading and setting up reference databases, it can take 12 hours or longer to run the example data here the first time. The example data are roughly 800 MB (the relatively large size for example data is so that MAGs are still recovered).

---


[toc]

---

## Installing conda, mamba, and genelab-utils
### conda
If conda is not already present, we recommend miniconda. Installers can be found from conda [here](https://conda.io/en/latest/miniconda.html), and if helpful, command-line installation is walked-through [here](https://astrobiomike.github.io/unix/conda-intro#getting-and-installing-conda).

### mamba
Once conda is installed on your system, we recommend installing [mamba](https://github.com/mamba-org/mamba#mamba), as it generally allows for much faster conda installations:

```bash
conda install -y -n base -c conda-forge mamba
```

### genelab-utils
The workflows are retrieved and meant to be run within this packages environment created here:

```bash
mamba create -y -n genelab-utils -c conda-forge -c bioconda -c defaults -c astrobiomike 'genelab-utils>=1.2.19'
```

The rest below expects to be done in the genelab-utils environment activated here:

```bash
conda activate genelab-utils
```

---

## Getting workflow and example data

```bash
# this downloads the workflow
GL-get-workflow MG-Illumina
```

```bash
# this grabs example data for 2 samples (about 800 MB)
GL-get-Illumina-metagenomics-test-data
```

---

## Modifying config.yaml and creating input samples file

```bash
# changing into the workflow directory
cd SW_MGIllumina*/

# setting the input reads directory variable
sed -i 's|../Raw_Sequence_Data/|../example-metagenomic-reads/|' config.yaml

  # or if on mac/Darwin system and the above error'd
# sed -i "" 's|../Raw_Sequence_Data/|../example-metagenomics-reads/|' config.yaml

# creating input file with unique sample IDs
printf "Sample-1\nSample-2\n" > unique-sample-IDs.txt
```

You must also set the primary location that will hold the reference databases in the config.yaml file. It is the "REF_DB_ROOT_DIR" variable.

## Running the workflow

As mentioned above, this workflow relies on some large reference databases. They will be installed and setup automatically by the workflow the first time it is run. All together, after downloaded and unpacked, they will take up about 240GB of storage, but they also may require up to 500GB during installation and initial un-packing. Largely due to database download and setup the first time, the first run of the workflow takes a while, likely greater than 12 hours. 

Running the workflow the first time will also create all needed environments, and then if run as shown below (specifying the `--conda-prefix` as shown), they will be re-used in future runs. This is expected to be executed within the genelab-utils environment.

### Standard execution

The below is one example command to execute the workflow:

```bash
snakemake --use-conda --conda-prefix ${CONDA_PREFIX}/envs -j 2 -p
```

### Telling snakemake to use slurm

There is info about [snakemake/slurm here](https://snakemake.readthedocs.io/en/stable/executing/cluster.html#executing-on-slurm-clusters), and a default location it checks for a slurm configuration file is `~/.config/snakemake/slurm/config.yaml`. Here is a template of what one of mine looks like:

```
cluster:
    mkdir -p slurm-logs &&
    if [ -z {wildcards} ]; then log_wildcard=""; else log_wildcard=$(echo "-{wildcards}" | sed 's/ID=//'); fi  &&
    sbatch --mem={resources.mem_mb} -c {resources.cpus} -J {rule} -o slurm-logs/{rule}${{log_wildcard}}-%j.log -e slurm-logs/{rule}${{log_wildcard}}-%j.log
use-conda:
    True
cores:
    50
jobs:
    10
printshellcmds:
    True
reason:
    True
rerun-incomplete:
    True
scheduler:
    greedy
latency-wait:
    60
default-resources:
    - cpus=1
    - mem_mb=2000
```

And here is an example execution telling snakemake to manage things with slurm:

```bash
snakemake --use-conda --conda-prefix ${CONDA_PREFIX}/envs --profile slurm
```

---

## Peeking at some outputs

From the location where the workflow was run:

**Gene-level function coverages**

```bash
head ../Assembly-based_Processing/combined-outputs/Combined-gene-level-KO-function-coverages.tsv | column -t | sed 's/^/# /'
# KO_ID   KO_function                                                                              Sample-1  Sample-2
# K00001  alcohol dehydrogenase [EC:1.1.1.1]                                                       0.0       105.02940000000001
# K00003  homoserine dehydrogenase [EC:1.1.1.3]                                                    0.0       102.3683
# K00005  glycerol dehydrogenase [EC:1.1.1.6]                                                      76.3986   0.0
# K00009  mannitol-1-phosphate 5-dehydrogenase [EC:1.1.1.17]                                       55.6388   5.1954
# K00010  myo-inositol 2-dehydrogenase / D-chiro-inositol 1-dehydrogenase [EC:1.1.1.18 1.1.1.369]  93.0      0.0
# K00012  UDPglucose 6-dehydrogenase [EC:1.1.1.22]                                                 88.2442   103.9793
# K00013  histidinol dehydrogenase [EC:1.1.1.23]                                                   82.0575   80.1397
# K00014  shikimate dehydrogenase [EC:1.1.1.25]                                                    173.8462  356.61289999999997
# K00015  glyoxylate reductase [EC:1.1.1.26]                                                       0.0       106.5199
```

**Recovered MAGs**

```bash
head ../Assembly-based_Processing/MAGs/MAGs-overview.tsv | column -t | sed 's/^/# /'
# Assembly        Total contigs  Total length  GC content  Maximum contig length  N50     L50  Num. contigs >= 10000  Num. contigs >= 50000  Num. contigs >= 100000  est. completeness  est. redundancy  est. strain heterogeneity  domain    phylum          class                order        family            genus             species
# Sample-2-MAG-2  132            5932921       71.61       305214                 69375   27   115                    40                     12                      98.58              1.57             41.67                      Bacteria  Proteobacteria  Alphaproteobacteria  Rhizobiales  Beijerinckiaceae  Methylobacterium  Methylobacterium aquaticum_B
# Sample-2-MAG-3  118            6399692       69.54       410159                 100140  19   96                     45                     20                      98.65              0.94             25.00                      Bacteria  Proteobacteria  Alphaproteobacteria  Rhizobiales  Beijerinckiaceae  Methylobacterium  NA
```

