Snakemake Hackathon

Event info

Hackathon date: 22.05 , 10-16
Location: online
Event page and registration: https://ssl.eventilla.com/event/snakemake_hack (published on 21.3)
Costs: free of charge
Max participants: currently 25, to be updated if needed
Collaborative document: https://siili.rahtiapp.fi/snakemake_hack (draft)

Organizer: CSC (Geoportti project)
Collaborators:

projects: CodeRefinery, EuroCC2
organizations: CSC, Aalto, ENCCS, PDC, UPPMAX …?
people:
- CSC: Antoni Golos, Laxmana Yetukuri, Samantha Wittke
- Aalto: Teemu Ruokolainen
- ENCCS: Yonglei Wang
- PDC (NAISS): Johan Hellsvik
- UPPMAX (NAISS): Diana Iusan
Registration for all via CSC event page system

Snakemake Hackathon

Post workshop email

Hei,

thank you so much for joining the Snakemake Hackathon today and asking your questions! And thanks also to all our instructors and helpers today :)
Do not worry if you could not join today after all, we might do this again in the future.

If you did join, please give us feedback on this event, either by adding it to the collaborative document(https://siili.rahtiapp.fi/snakemake_hack?view#-This-is-a-pilot-event) or answering this e-mail. It is very important to us, to further develop the materials and format.

All materials will stay available for the foreseeable future: https://coderefinery.github.io/snakemake_hackathon/
The Q&A document will also stay available for a while here: https://siili.rahtiapp.fi/snakemake_hack (but please do not add any new questions, as we might not see them there, use cluster support channels instead)

If you have any questions about running Snakemake on your cluster, please contact the respective support help line:
- CSC: servicedesk@csc.fi or Weekly research support coffees (https://ssl.eventilla.com/usersupportcoffee/EN), Wednesdays at 2 p.m. where you can directly discuss with our specialists!
- LUMI: Contact LUMI user support team: https://lumi-supercomputer.eu/user-support/need-help/
- UPPMAX: Support web interface: https://supr.naiss.se/support/

If you are a Puhti user were affected by the SLURM issues today: We are sorry for the inconvenience. You may contact us via above channels if you encounter any issues later, and we can also agree on a zoom session, if needed.

This event has been brought to you by Geoportti (https://www.geoportti.fi/) , CodeRefinery (https://www.coderefinery.org/) and partners.

Hope to see you again in future events!

Kind regards,
Samantha and Snakemake hackathon team

Materials

rendered: https://coderefinery.github.io/snakemake_hackathon/
source: https://github.com/coderefinery/snakemake_hackathon

Schedule

Latest schedule: https://siili.rahtiapp.fi/snakemake_hack?view#📅-Agenda

Participant stats

Current counts:

Home organization nationality:

UK:2
Fin: 19
Swe:12
TR: 2

HPC experience:

no: 3
beginner: 22
advanced: 12

HPC access:

Aalto Triton: 0
CSC Mahti: 1
CSC Puhti: 17
LUMI: 3
UPPPMAX: 9 (bianca, rackham/snowy)
Other 5
- UK: Oxford Arc: 2
- NAISS C3SE: 1
- VTT internal: 2
- other (person from Lund, 1 from Turkey)

Snakefile:

No: 21
WIP: 7
ready: 7
Don't know: 1

Planning meeting 13.05.24

4 people from waiting list accepted
Participant email check
UPPMAX Snakemake usage presentation
- snakemake module
- conda?
  - users can do own
- hyperqueue?
  - no
  - could be provided
- interactive session
CSC
- plain module installation
  - good for use with other modules
  - plain pip installations: pygment, snakemake, snakemake slurm and generic plugin
- container wrapper for conda based installations (tykky)
- container based installation
- launching jobs
  - native snakemake cluster integration: one job per rule may be overkill
  - hyperqueue
General Snakemake on cluster
- general portability recommendations
- general best practices
- one job interactive
- slurm integration
General room?
- Other specialists
Dummy example:
- Hyperqueue available on other clusters?
- Diana to run on Uppmax as test

CSC Planning meeting 08.05.24

HPC refresher materials: https://coderefinery.github.io/TTT4HPC_resource_management/#
- Does this cover everything needed?
Send HPC refresher week before the course, have first hour Q&A, hangout
LUMI environment set up
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
Mahti environment not set up, ask participant to switch to Puhti
High Throughput example with Hyperqueue and Snakemake (for people without own workflow), modified from CSC docs: https://github.com/csc-training/csc-env-eff/blob/11-workflows-dev/part-2/workflows/snakemake-ht.md
(WIP) CSC- Snakemake intro, slides: https://a3s.fi/CSC_training/snakemake_hackathon.html

Participant e-mail draft

To be sent 14.5

Dear all,

thank you for your interest in the Snakemake hackathon.

Joining the event

This event will be fully online in Zoom.

Join Zoom Meeting: https://cscfi.zoom.us/j/61757837253
Meeting ID: 617 5783 7253

Agenda

All times in EEST (Helsinki time):

10-10.30 Supercomputing concept refresher Q&A
10.30-11.30 Intro to Snakemake
11.30-12 Snakemake on HPC (first general, then cluster specific)
12-13 Lunch break
13-15 Hackathon (cluster specific breakoutrooms for CSC Puhti + LUMI, UPPMAX and one general room)

You can join any session of interest to you.
Please note that due to the feedback during registration, we will not have a lecture on supercomputing concepts, but have a Q&A/discussion session about materials provided below in the morning.
The idea for the hackathon part is that you work on your own workflow and ask help from specialists where needed. You can also check out our toy example.

Requirements

Please read this material as HPC concept refresher: https://coderefinery.github.io/TTT4HPC_resource_management/#
- We have reserved 30 min in the beginning of the course for questions related to this material
For the hands-on hackathon part, you need to be able to run stuff on a cluster:
- For CSC clusters this means, that you need an account and a project with billing units.
  - If you do not have a project, please send us your CSC username until 20.5 by answering to this email.
- For UPPMAX clusters this means, that you need an account.
  - You will be added to a course project in the beginning of the hackathon session

Note on support

For participants not using Aalto Triton, CSC Puhti/Mahti, LUMI, ENCCS or UPPMAX clusters, we unfortunately cannot provide cluster specific support.
You may still try it out on your cluster/own laptop and ask about Snakemake related stuff though :)

Interaction during the event

We will use a collaborative document to ask, answer and collect all questions and feedback during the event: https://siili.rahtiapp.fi/snakemake_hack?view

Further information

Please refer to the event page for general information about the event: https://ssl.eventilla.com/snakemake_hack

In case of any questions, please answer to this email.

Looking forward to virtually meet you on 22.5!

Kind regards,
Samantha and the Snakemake hackathon team

Planning meeting 30.4.24

HPC refresher as materials beforehand, offer Q&A
Antoni Snakemake intro
- What is snakemake
- When does it make sense to use Snakemake, when not
- Implementation of workflow with Snakemake
Snakemake on HPC (Laxmana)
- cluster specific short presentations in breakoutroom
- CSC: Mahti, Puhti, LUMI
Puhti: LY has project, can invite
Week before:
- need Puhti project?

Next catch up 8.5

LY: check Snakemake on LUMI, install as module, Snakemake on Puhti/Mahti/LUMI presentation
SW: update eventilla with HPC refresher as materials, link to materials
AG: Snakemake intro

Planning meeting 4.4.24

Idea and plan for the Hackathon
- morning: lectures, afternoon: hands-on
HPC concepts; difference between clusters? (Dardel similar to LUMI)
- SLURM?
  Image Not Showing Possible Reasons
  - The image file may be corrupted
  - The server hosting the image is unavailable
  - The image path is incorrect
  - The image format is not supported
  Learn More →
- "fair share"? job priorities
  - about same
- conda
  - some allow, some prefer containerized, CSC: tykky
- container
  - singularity/apptainer
- local scratch
Snakemake on your cluster?
- Do we want breakout rooms already here?
- CSC:
  - doc: https://docs.csc.fi/apps/snakemake/
  - tutorial: https://docs.csc.fi/support/tutorials/snakemake-puhti/
  - own installation via "container wrapper": https://docs.csc.fi/computing/containers/tykky/
- LUMI:
  - similar to CSC setup
  - …?
- Dardel
  - same computer model as LUMI. Python and/or Snakemake configurations might differ
- UPPMAX
- Triton
- ENCCS

Have short intro for each cluster in beginning of hackathon session in breakoutrooms

Example case for hackathon (people without own project)? Close to "real life" but not Science specific.
- fall back solution: word count from HPC carpentries
- add your example here:
  - ..
  - ..
Involvement interest (all, add your initials)
- lectures
  - LY
  - AG
- hackathon
  - DI
  - YW
  - JH
- advertising
  - all
Advertisement:
- CSC April training newsletter (SW)
  Image Not Showing Possible Reasons
  - The image file may be corrupted
  - The server hosting the image is unavailable
  - The image path is incorrect
  - The image format is not supported
  Learn More →
- CSChpc twitter (SW)
  Image Not Showing Possible Reasons
  - The image file may be corrupted
  - The server hosting the image is unavailable
  - The image path is incorrect
  - The image format is not supported
  Learn More →
- Geoportti (SW)
- EuroCC (YW)
- CodeRefinery (SW)
- TTT4HPC Day4 course (TR)
- Aalto SciComp (TR)
- NAISS
  Image Not Showing Possible Reasons
  - The image file may be corrupted
  - The server hosting the image is unavailable
  - The image path is incorrect
  - The image format is not supported
  Learn More →
- ENCCS

TODO

ASAP:
- advertise (use list above)
- start material preparation (SW to coordinate CSC internal first, before opening for comnments/additions from others)
Beginning of May:
- prepare environment / module on each system
- add cluster specific tabs to materials (all)
- prepare short cluster specific intros (all)
Mid May:
- Course project (on each system)
  - invite participants
- course resource reservation (on each system)