<section data-background-image="https://raw.githubusercontent.com/MaxUlysse/maxulysse.github.io/master/assets/img/svg/green_white_bg.svg" data-background-opacity=0.9 >
<a href="https://www.nf-co.re"><img src="https://raw.githubusercontent.com/nf-core/logos/master/byte-size-logos/bytesize-darkbg.svg" width="65%"><img></a>
# \#10: Making a new institutional profile
James Fellows Yates / <img src="https://openmoji.org/data/color/svg/E040.svg" width=50><img src="https://openmoji.org/data/color/svg/E045.svg" width=50> @jfy133
LMU München / MPI-EVA
---
# Overview
- Benefits nf-core institutional profiles
- Information to gather before writing
- Step-by-step example of profile
- How to test a draft profile
---
# Today
> 🏫 **_Global_ institutional profile**: a Nextflow configuration file usable by **all** nf-core pipelines to work efficiently on institutional-level cluster(s). Stored on [nf-core/configs](https://github.com/nf-core/configs).
- _Pipeline institutional profiles: future bytesize_
---
# Why use?
- 🚀 efficiency: computing resource, time
- 🛄 portable: assists reproducibility
- ⌚ saves time: write it once, everyone benefits
---
# Recap
_General info about nf-core & configs: [bytesize #2](https://nf-co.re/events/2021/bytesize-2-configs)_
> 📄 Nextflow configuration file: simple text file containing a set of properties (parameters, etc.)
- Different levels!
- `./nextflow.config`
- `$HOME/.nextflow/config`
- `-c`
- **nf-core profiles: nf-core/configs**
<aside class="notes">
nf-core/profiles: download on run-time from from nf-core/configs!
- _Download for offline use: [nf-core/tools](https://www.nf-co.re/tools)_
</aside>
---
Example:
> ```bash
> nextflow run nf-core/eager -profile uppmax
> ```
>
> ```bash
> nextflow run nf-core/eager -profile shh,sdag
> ```
>
> ```bash
> nextflow run nf-core/mag -profile shh,sdag
> ```
:warning: order matters: furthest right has precedence!
<aside class="notes">
Institutional profiles apply for ALL pipelines
Can chain multiple together: institution, specific-cluster, project, run!
Important Last one takes precedence, each will overwrite same parameters in the profile to the left of it
Therefore, normally institution will be _first_
</aside>
---
# Preparation!
---
# Topics to cover
1. 📛 names
2. 🛑 resource limits
3. 📆 scheduling systems
4. 📦 containers
---
# Names
- 📛 Do you have recognisable and descriptive name?
- Short is good (abbreviations OK!)
- Precise but compact
<aside class="notes">
Always double check you have permission to do so - very occasionally some clusters may which to keep aspects private
Precise: easily distinguisable from other clusters
But compact: if one institution has multiple clusters, use an umbrella name, and can make 'internal' profiles for each cluster (see later)
</aside>
---
# Resource limits
- To effectively use nf-core retry ✨:
- 💾 Largest node's memory
- 💻 Largest node's CPU
- ⌚ Longest queue's walltime
- 📁 Scratch usage?
---
# Scheduling systems
- For Nextflow to submit for you:
- 📅 What scheduler (check [Nextflow docs](https://www.nextflow.io/docs/latest/config.html#scope-executor)!)?
- 🐍 Queues/partitions?
- 🛑 Submission limits?
- 🛠️ Additional configurations (e.g. `module load`)?
---
# Containers
- For robust reproducibility:
- 📦 Which container engine (check [nf-core docs](https://nf-co.re/usage/configuration#basic-configuration-profiles)!)?
- 📁 Common cache locations?
- 🛠️ Additional parameters?
---
# Writing!
---
# To start
- 🍴 Fork [nf-core/configs](https://github.com/nf-core/configs)
- 🌲 Make a branch
- 📖 Open [Nextflow documentation](https://www.nextflow.io/docs/latest/index.html)
---
# To start
- ✍🏽 Create two new files:
- `conf/<your_cluster>.conf`
- `docs/<your_cluster>.md`
- 📛 Add profile name to:
- `nfcore_custom.config`
- `README.md`
- `.github/workflows/main.yml`
---
# Scope: `params`
```groovy
params {
config_profile_description = '<cluster_name> cluster profile provided by nf-core/configs.'
config_profile_contact = '<your_name> (<your_github_handle>)'
config_profile_url = 'https://<institutional_url>.com'
max_memory = 2.TB
max_cpus = 128
max_time = 720.h
igenomes_base = '/<path>/<to>/igenomes/' // optional!
}
```
In `conf/<your_cluster>.conf`
---
# Scope: `process`
Simple example
```groovy
process {
executor = 'slurm'
maxRetries = 2
}
```
Complex example
```groovy
process {
executor = 'sge'
queue = { task.time <= 2.h ? 'short' : task.time <= 24.h ? 'medium': 'long'
maxRetries = 2
clusterOptions = { '-l h_vmem=${task.memory.toGiga()}G' }
}
```
In `conf/<your_cluster>.conf`
---
# Scope: `executor`
```groovy
executor {
queueSize = 8
submitRateLimit = '10 sec'
}
```
In `conf/<your_cluster>.conf`
---
# Scope: `container`
```groovy
singularity {
enabled = true
autoMounts = true
cacheDir = '/<path>/<to>/<your>/<image_cache>'
}
```
In `conf/<your_cluster>.conf`
---
# Scope: `profiles`
```groovy
<...>
profiles {
red {
params {
config_profile_description = '<your_institution_name> 'red' cluster cluster profile provided by nf-core/configs.'
max_memory = 2.TB
max_cpus = 128
max_time = 720.h
}
}
blue {
params {
config_profile_description = '<your_institution_name> 'blue' cluster profile provided by nf-core/configs.'
max_memory = 256.GB
max_cpus = 64
max_time = 24.h
}
}
}
```
(_Or use [`hostnames`](https://github.com/nf-core/configs/blob/fb40b27eaba94bd0c43b2f234b0bea3430ff7112/conf/uppmax.config#L30-L42)_ )
In `conf/<your_cluster>.conf`
---
# Documentation!
---
# What to include?
- 🗺️ Where the cluster is based
- 🛑 Summary of parameters
- e.g. resource limits, queues
- 👨💻 Instructions for user-level configuration
- e.g. cache directories
- 🖧 Available 'sub'-profiles
In `docs/<your_cluster>.md`
<aside class="notes">
Where: in case same/similar name of cluster at other institution
</aside>
---
# Testing and submission!
---
# Test profile from your fork
```bash
nextflow run nf-core/<fav_pipeline> \
-profile <your_cluster_name>,test \
--custom_config_base 'https://raw.githubusercontent.com/<your_github_user>/configs/<your_branch>'
```
> ⚠️ Expect trial and error!
<aside class="notes">
Things to check when testing: pipeline fail?
- Profile description displayed in run summary header?
- Jobs sent to expected queue/present in scheduler log?
- Container stored in specified cache directory
</aside>
---
# Submit to nf-core/configs!
- 📑 Make a PR to nf-core/configs
- 📢 On slack: #request-review
- 🥳 Once approved, merge, and publicise!
- 👩🏼💻 From now on:
```bash
nextflow run nf-core/<fav_pipeline> \
-profile <your_cluster_name>
<...>
```
---
<section .slide: data-background-image="https://raw.githubusercontent.com/MaxUlysse/maxulysse.github.io/master/assets/img/svg/green_white_bg.svg" data-background-opacity=0.9 >
## Need help?
Repository: [`nf-core/configs`](https://github.com/nf-core/configs)
Tutorial: [`https://nf-co.re/usage/usage_tutorials`](https://nf-co.re/usage/usage_tutorials)
Chat: [`https://nf-co.re/join`](https://nf-co.re/join) <img src="https://cdn.brandfolder.io/5H442O3W/at/pl546j-7le8zk-6gwiyo/Slack_Mark.svg" width=10%></img>#configs
## Next Bytesize
Development environments & workflows
May 4th 2021, 13:00 CEST
<p align="center">
Follow nf-core on
<a href="https://www.twitter.com/nf_core"><img src="https://openmoji.org/data/color/svg/E040.svg" width=6%></a>
<a href="https://github.com/nf-core"><img src="https://openmoji.org/data/color/svg/E045.svg" width=6%></a>
<a href="https://www.youtube.com/c/nf-core"><img src="https://openmoji.org/data/color/svg/E044.svg" width=6%></a>
</a>
</p>
<style>
.reveal section img { background:none; border:none; box-shadow:none; }
body {
background-image: url(https://raw.githubusercontent.com/nf-core/logos/master/nf-core-logos/nf-core-logo-square.svg);
background-size: 7.5%;
background-repeat: no-repeat;
background-position: 3% 96%;
background-color: #181a1b;
}
.reveal body {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal p {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal h1 {
font-family: 'Roboto', sans-serif;
font-style: bold;
font-weight: 400;
color: white;
font-size: 62px;
}
.reveal h2 {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal h3 {
font-family: 'Roboto', sans-serif;
font-style: italic;
font-weight: 300;
color: white;
}
.reveal p {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal li {
font-family: 'Roboto', sans-serif;
font-weight: 300;
color: white;
}
.reveal pre {
background-color: #272822 !important;
display: inline-block;
border-radius: 7px;
color: #aaaba9;
}
.reveal pre code {
color: #aaaba9;
background-color: #272822;
font-size: 100%;
}
.reveal code {
background-color: #272822;
font-size: 75%;
}
.reveal .progress {
color: #24B064;
}
.reveal .controls button {
color: #24B064;
}
.reveal blockquote {
display: block;
position: relative;
width: 90%;
margin: 20px auto;
padding: 5px;
background: rgba(255, 255, 255, 0.05);
box-shadow: 0px 0px 2px rgb(0 0 0 / 20%);
}
</style>
{"title":"nf-core/bytesize - Making a new institutional profile","tags":"nf-core, bytesize, open-source, talk"}