<section data-background-image="https://raw.githubusercontent.com/MaxUlysse/maxulysse.github.io/master/assets/img/svg/green_white_bg.svg" data-background-opacity=0.9 > <a href="https://www.nf-co.re"><img src="https://raw.githubusercontent.com/nf-core/logos/master/byte-size-logos/bytesize-darkbg.svg" width="65%"><img></a> # \#10: Making a new institutional profile James Fellows Yates / <img src="https://openmoji.org/data/color/svg/E040.svg" width=50><img src="https://openmoji.org/data/color/svg/E045.svg" width=50> @jfy133 LMU MΓΌnchen / MPI-EVA --- # Overview - Benefits nf-core institutional profiles - Information to gather before writing - Step-by-step example of profile - How to test a draft profile --- # Today > 🏫 **_Global_ institutional profile**: a Nextflow configuration file usable by **all** nf-core pipelines to work efficiently on institutional-level cluster(s). Stored on [nf-core/configs](https://github.com/nf-core/configs). - _Pipeline institutional profiles: future bytesize_ --- # Why use? - πŸš€ efficiency: computing resource, time - πŸ›„ portable: assists reproducibility - ⌚ saves time: write it once, everyone benefits --- # Recap _General info about nf-core & configs: [bytesize #2](https://nf-co.re/events/2021/bytesize-2-configs)_ > πŸ“„ Nextflow configuration file: simple text file containing a set of properties (parameters, etc.) - Different levels! - `./nextflow.config` - `$HOME/.nextflow/config` - `-c` - **nf-core profiles: nf-core/configs** <aside class="notes"> nf-core/profiles: download on run-time from from nf-core/configs! - _Download for offline use: [nf-core/tools](https://www.nf-co.re/tools)_ </aside> --- Example: > ```bash > nextflow run nf-core/eager -profile uppmax > ``` > > ```bash > nextflow run nf-core/eager -profile shh,sdag > ``` > > ```bash > nextflow run nf-core/mag -profile shh,sdag > ``` :warning: order matters: furthest right has precedence! <aside class="notes"> Institutional profiles apply for ALL pipelines Can chain multiple together: institution, specific-cluster, project, run! Important Last one takes precedence, each will overwrite same parameters in the profile to the left of it Therefore, normally institution will be _first_ </aside> --- # Preparation! --- # Topics to cover 1. πŸ“› names 2. πŸ›‘ resource limits 3. πŸ“† scheduling systems 4. πŸ“¦ containers --- # Names - πŸ“› Do you have recognisable and descriptive name? - Short is good (abbreviations OK!) - Precise but compact <aside class="notes"> Always double check you have permission to do so - very occasionally some clusters may which to keep aspects private Precise: easily distinguisable from other clusters But compact: if one institution has multiple clusters, use an umbrella name, and can make 'internal' profiles for each cluster (see later) </aside> --- # Resource limits - To effectively use nf-core retry ✨: - πŸ’Ύ Largest node's memory - πŸ’» Largest node's CPU - ⌚ Longest queue's walltime - πŸ“ Scratch usage? --- # Scheduling systems - For Nextflow to submit for you: - πŸ“… What scheduler (check [Nextflow docs](https://www.nextflow.io/docs/latest/config.html#scope-executor)!)? - 🐍 Queues/partitions? - πŸ›‘ Submission limits? - πŸ› οΈ Additional configurations (e.g. `module load`)? --- # Containers - For robust reproducibility: - πŸ“¦ Which container engine (check [nf-core docs](https://nf-co.re/usage/configuration#basic-configuration-profiles)!)? - πŸ“ Common cache locations? - πŸ› οΈ Additional parameters? --- # Writing! --- # To start - 🍴 Fork [nf-core/configs](https://github.com/nf-core/configs) - 🌲 Make a branch - πŸ“– Open [Nextflow documentation](https://www.nextflow.io/docs/latest/index.html) --- # To start - ✍🏽 Create two new files: - `conf/<your_cluster>.conf` - `docs/<your_cluster>.md` - πŸ“› Add profile name to: - `nfcore_custom.config` - `README.md` - `.github/workflows/main.yml` --- # Scope: `params` ```groovy params { config_profile_description = '<cluster_name> cluster profile provided by nf-core/configs.' config_profile_contact = '<your_name> (<your_github_handle>)' config_profile_url = 'https://<institutional_url>.com' max_memory = 2.TB max_cpus = 128 max_time = 720.h igenomes_base = '/<path>/<to>/igenomes/' // optional! } ``` In `conf/<your_cluster>.conf` --- # Scope: `process` Simple example ```groovy process { executor = 'slurm' maxRetries = 2 } ``` Complex example ```groovy process { executor = 'sge' queue = { task.time <= 2.h ? 'short' : task.time <= 24.h ? 'medium': 'long' maxRetries = 2 clusterOptions = { '-l h_vmem=${task.memory.toGiga()}G' } } ``` In `conf/<your_cluster>.conf` --- # Scope: `executor` ```groovy executor { queueSize = 8 submitRateLimit = '10 sec' } ``` In `conf/<your_cluster>.conf` --- # Scope: `container` ```groovy singularity { enabled = true autoMounts = true cacheDir = '/<path>/<to>/<your>/<image_cache>' } ``` In `conf/<your_cluster>.conf` --- # Scope: `profiles` ```groovy <...> profiles { red { params { config_profile_description = '<your_institution_name> 'red' cluster cluster profile provided by nf-core/configs.' max_memory = 2.TB max_cpus = 128 max_time = 720.h } } blue { params { config_profile_description = '<your_institution_name> 'blue' cluster profile provided by nf-core/configs.' max_memory = 256.GB max_cpus = 64 max_time = 24.h } } } ``` (_Or use [`hostnames`](https://github.com/nf-core/configs/blob/fb40b27eaba94bd0c43b2f234b0bea3430ff7112/conf/uppmax.config#L30-L42)_ ) In `conf/<your_cluster>.conf` --- # Documentation! --- # What to include? - πŸ—ΊοΈ Where the cluster is based - πŸ›‘ Summary of parameters - e.g. resource limits, queues - πŸ‘¨β€πŸ’» Instructions for user-level configuration - e.g. cache directories - πŸ–§ Available 'sub'-profiles In `docs/<your_cluster>.md` <aside class="notes"> Where: in case same/similar name of cluster at other institution </aside> --- # Testing and submission! --- # Test profile from your fork ```bash nextflow run nf-core/<fav_pipeline> \ -profile <your_cluster_name>,test \ --custom_config_base 'https://raw.githubusercontent.com/<your_github_user>/configs/<your_branch>' ``` > ⚠️ Expect trial and error! <aside class="notes"> Things to check when testing: pipeline fail? - Profile description displayed in run summary header? - Jobs sent to expected queue/present in scheduler log? - Container stored in specified cache directory </aside> --- # Submit to nf-core/configs! - πŸ“‘ Make a PR to nf-core/configs - πŸ“’ On slack: #request-review - πŸ₯³ Once approved, merge, and publicise! - πŸ‘©πŸΌβ€πŸ’» From now on: ```bash nextflow run nf-core/<fav_pipeline> \ -profile <your_cluster_name> <...> ``` --- <section .slide: data-background-image="https://raw.githubusercontent.com/MaxUlysse/maxulysse.github.io/master/assets/img/svg/green_white_bg.svg" data-background-opacity=0.9 > ## Need help? Repository: [`nf-core/configs`](https://github.com/nf-core/configs) Tutorial: [`https://nf-co.re/usage/usage_tutorials`](https://nf-co.re/usage/usage_tutorials) Chat: [`https://nf-co.re/join`](https://nf-co.re/join) <img src="https://cdn.brandfolder.io/5H442O3W/at/pl546j-7le8zk-6gwiyo/Slack_Mark.svg" width=10%></img>#configs ## Next Bytesize Development environments & workflows May 4th 2021, 13:00 CEST <p align="center"> Follow nf-core on <a href="https://www.twitter.com/nf_core"><img src="https://openmoji.org/data/color/svg/E040.svg" width=6%></a> <a href="https://github.com/nf-core"><img src="https://openmoji.org/data/color/svg/E045.svg" width=6%></a> <a href="https://www.youtube.com/c/nf-core"><img src="https://openmoji.org/data/color/svg/E044.svg" width=6%></a> </a> </p> <style> .reveal section img { background:none; border:none; box-shadow:none; } body { background-image: url(https://raw.githubusercontent.com/nf-core/logos/master/nf-core-logos/nf-core-logo-square.svg); background-size: 7.5%; background-repeat: no-repeat; background-position: 3% 96%; background-color: #181a1b; } .reveal body { font-family: 'Roboto', sans-serif; font-weight: 300; color: white; } .reveal p { font-family: 'Roboto', sans-serif; font-weight: 300; color: white; } .reveal h1 { font-family: 'Roboto', sans-serif; font-style: bold; font-weight: 400; color: white; font-size: 62px; } .reveal h2 { font-family: 'Roboto', sans-serif; font-weight: 300; color: white; } .reveal h3 { font-family: 'Roboto', sans-serif; font-style: italic; font-weight: 300; color: white; } .reveal p { font-family: 'Roboto', sans-serif; font-weight: 300; color: white; } .reveal li { font-family: 'Roboto', sans-serif; font-weight: 300; color: white; } .reveal pre { background-color: #272822 !important; display: inline-block; border-radius: 7px; color: #aaaba9; } .reveal pre code { color: #aaaba9; background-color: #272822; font-size: 100%; } .reveal code { background-color: #272822; font-size: 75%; } .reveal .progress { color: #24B064; } .reveal .controls button { color: #24B064; } .reveal blockquote { display: block; position: relative; width: 90%; margin: 20px auto; padding: 5px; background: rgba(255, 255, 255, 0.05); box-shadow: 0px 0px 2px rgb(0 0 0 / 20%); } </style>
{"metaMigratedAt":"2023-06-15T22:47:52.916Z","metaMigratedFrom":"YAML","title":"nf-core/bytesize - Making a new institutional profile","breaks":true,"contributors":"[{\"id\":\"6357fb8d-b0f8-4b7d-9fbe-6d813ad67700\",\"add\":13826,\"del\":4145}]"}
    2752 views