BioMonth 2021

tags: `bioinformatics`

This is the common "notebook" for the "BioMonth" organised in March 2021 at CSC -IT center for Science. Course page in e-Lena platform

This is the place to ask questions about the workshop content! We use the Zoom chat only for posting links, reporting Zoom problems and such.

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Hint: HackMD is great for sharing information in this kind of courses, as the code formatting is nice & easy with MarkDown! Just add 3 ticks (`) for thecode blocks. Otherwise, it's like a Google doc: it allows simultaneous editing.

BioMonth 2021

Code of conduct

We strive to follow the Code of Conduct developed by The Carpentries organisation to foster a welcoming environment for everyone. In short:

Use welcoming and inclusive language
Be respectful of different viewpoints and experiences
Gracefully accept constructive criticism
Focus on what is best for the community
Show courtesy and respect towards other community members

TO DO before the course

Please make sure you have accepted the terms of use for Puhti and Allas in my.csc.fi for the course project (2002389, "BioMonth Puhti Allas")!
Check & test that you have a functional terminal connection (https://docs.csc.fi/computing/connecting/, graphical connection not needed)
Login to e-Lena course page (https://e-learn.csc.fi/course/view.php?id=57) and self-register to the course (registration key was sent to you via e-mail)
Check the Zoom instructions below

Zoom instructions

Link to the zoom room was sent to you via e-mail
Please arrive 5-10 minutes before to test your microphone setup.
During the course, please remember to always mute your microphone when you are not speaking.
Please use a headset in order to avoid echo (a simple phone headset is just fine).
You can find all the controls (mic, video, chat, screen sharing) at the bottom of the Zoom window (when you bring your mouse there).
You can use the chat box for questions and comments, but please make sure you reply to "all panelists and participants" instead of just "all panelists", which is often the default.
If you have a spoken question/comment, please use the "raise hand" button: we will then give the floor (and microphone rights) to you.
Note: for questions and answers about the course topics, we will be using this living document

Image Not Showing Possible Reasons
The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported
Learn More →

Q & A

You can type your questions here. We will answer them, and this document will store the answers for you for later use!

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

General and practical matters

Q0: I have difficulty pasting my questions into HackMD. Do you have some instructions on how to write here?
- A: Can you see these three icons on top left corner, next to HackMD text? There’s pencil, this side-by-side symbol, and an eye. In eye view, you can’t edit, you are just viewing. The other two reveal the markdown (MD) version of the page, which you can edit. I find it easiest to edit with the side-by-side view.

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Hint: You can also apply styling from the toolbar at the top

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

of the editing area.

Question 1: What is the schedule for today 16 March?
- Thanks for asking! Forgot to mention this! :) So we have lectures & exercises until 12:00, but we will keep a bit longer "bio-break" (15min) in between. Exercise sessions are hands-on, and a good time to stretch a bit, get a cup of coffee, etc… too :) -MariaL
Q13: I cannot access most of the slides from the course page. These are marked with an earth logo.
- A: Try to use the arrow keys or space bar to switch the slide! (Thanks for sharing this info, we need to add this tip there below the slides!)
Q17: Slides available after the course?
- A: Yes, you can access e-lena after the course as well! You can download the slides from there (right click and save), and the exercises (and slides, in one format) are available in github: https://github.com/CSCfi/csc-env-eff You can also convert the html slides to pdf by adding '?print-pdf' at the end of html file like this : https://a3s.fi/CSC_training/03_disk_areas.html?print-pdf
  and then type: control/cmd+p (on mac, cmd+p works)
Q18: Course certificate?
- A: Sure, send Maria an e-mail in case you need a course certificate!

Using Puhti

module load python-data
start-jupyter-server

If you have SSH keys set-up with Puhti, you can just follow the instructions that are displayed once jupyter is started. In case you use password to login to Puhti follow this ssh-tunneling process:

ssh -l <username> -L 8888:localhost:8888 puhti-login1.csc.fi    # Issue this command while being on local machine
                                                                # change port number if notebook is exposed on different port (default port is 8888 here); 
                                                                # choose login1 or login2 node 
                                                                                
ssh -l <username>  -L 8888:localhost:8888 <username>@$HOSTNAME       # Issue this command on login node; $HOSTNAME is compute node attached in batch job
                                                                     # hostname  of compute node attached to jupyter notebook

Point your browser to http://localhost:8888/?token=tokenkey (copy and paste this url from the instructions displayed once jupyter is launched)
If you want to use the server version, we recommend you set up the ssh-keys. Later this year, another setup called Open on Demand will at least partially provide another solution for this.

Data management

Question/comment 1: I like how the content in the RDM kit is organized as a reference manual. Nice example, perhaps something to adopt in a university web page as well?
- Answer: Great, thanks! Feedback is very welcome :)
Question 2: To clarify a detail about the informed consent. If we want to e.g. process data outside Finland although the consent restricts analysis to Finland. Is it enough to inform the patient or should we get new written consent?
- Answer: Francesca will talk to a colleague, and returns to this. It is a good idea to ask from your organisations legal personnel!
Question 3: How about the generation of the sequencing data? (Sequencing provider outside the lab)
- Answer: They are a service provider, so there might already be an agreement between institutions (provider and institution). Good point!
Question 4: Can a private key ever be recovered?
- A: No, if you lose your password, there is no way to de-encrypt the data. Be careful! :) Using our public key is a solution, as then there is a backup with us. There's a very strict policy in hosting our public key on our side.
Question 5: How will the encryption work in SD Connect, the tool to store sensitive data in Allas?
- A: For now, you need to do it yourself, later on it will be automated. Timeline is unclear still, we are working on this!
Question 6: Is crypt4gh available in Puhti?
- A: It is available in Puhti already, BUT, you are not allowed to import your sensitive data to Puhti un-encrypted. It is not yet possible to analyse sensitive data in HPC (Puhti, Mahti). You need to have the encryption tool installed locally, on your own computer. Here you can download the GUI: https://github.com/CSCfi/crypt4gh-gui/release

module load biokit
module load biopythontools
crypt4gh -h

Using Allas

Question 1: If you do not remember which protocol (swift/s3) you have used, can it be determined from the allas object?
- Answer: Not necessarily, but when not possible, it's not that much a problem. Small files you can use with both protocols. Bucket names can help: a-commands give by default the projectnumber-puhti-filename names. A-commands packages/compresses the files into one tar package.
Question 2: What if the bucket doesn't exist in Allas, when using rclone copy?
- Answer: The command creates the bucket.
Question 3: Is there the 5G limit with a-put command?
- Answer: No. Kimmo demonstrated what happens with a >5GB file: the file is split into several pieces, segments. Two buckets are created: In the main bucket, there's info where the segments are (size=0), and the actual segments are in the segments folder/bucket. Luckily, you don't really have to worry about this :)
Q4: Does the connection to allas get interrupted automatically after 8h? So is there a timer so we can check for how long we have been logged on to allas?
- Answer: Yes. This might cause problems when moving large files (>500GB). Tip: the -k option = session remembers your password, so the Allas connection can be automatically re-created, when new a-put is launched. This slows down the process, so don't use when necessary! For batch jobs this is handy.
Q5: Is there a python/R interface for a-tools?
- Answer: No.
Q6: Would the other big files (>5G) go to the same segment-bucket as the previously moved ones?
- Answer: This depends on the bucket name, which you define when uploading the data.
Q6: Can I directly transfer all data in my poject_200XXX direcotry using a-put /scracth/Project_200XXX?
- A: Good question! Don't try to move data within Puhti with Swift protocol/rclone. You need to pull it back to Puhti and then again to Allas.
Q7: No autocomplete in Allas? Tricky to navigate!
- A: Nope, it is not a filesystem so there's no file hierarchy (even thou the filenames LOOK like file paths!).
Q8: What does -P mean in command rclone -P?
- A: "Follow progress" = what is the transfer rate and how much have been moved already. BTW, commands have the help pages, -h! This is helpful when you don't know what a parameter means
Q9: When running backups, is it typically OK to do this from the login node? Do we need to go to compute node?
- A: Normally, it's ok to use login node for this. If there's a very large object to be moved, the compression starts to require quite a lot CPU time, so then the interactive compute node would be the nice thing to do :) More resources there.
Q10: About publicity: Allas is not 'eternal' link, so it cannot be used as a way to share data in publications, or can it?
- A: Yes and no :) Don't use the link in publications. It is valid as long as you have the project in Allas. This is a good way to share data with collaborators. DOI and such needed when really publishing data, check IDA at that point!
Q11: What does -b mean in command a-put pythium/* -b xxxx-genomes-ap?
- A: Same as --bucket, "Define a name of the bucket into which the data is uploaded". a-put -h helps when wondering what these options mean
Q12: How can one save the big result files of sbatch-runs directly to Allas? Should the compression be avoided, if the result files are then later shared with other people in other institutions (published)?
- A: Yes, you can make the sbatch runs store the data to Allas. -k is the trick here, before you launch the batch job. The you can use a-put with options –nc or –asis if you dont want to use the automatic compression or packing.
Q12: I wonder if there is a command to empty the console in PuTTY?
- A: Try clear
Q13: To download data from outside CSC to Allas, how it is done? Just replacing the path with 'pythium'( as the command was 'a-put pythium')?
- A: If you have user account, you can use the web interface (or Cyberduck). Or you can install a-tools or rclone to your environment, if the need is more continuous.
Q14: Can you just upload file from your computer to Allas via cPouta?
- A: Yes. You just need to do it one by one. We are developing the interface, so that it would be possible to upload a folder.
Q15: Is there a way to copy files from Funet FileSender link straight to Allas, i.e. without the need to first copy to your own computer?
- A: You can use rclone copyurl for that. Copy the download link from FunetFilesender. Then use it in quotation marks with rclone copyurl. Like:
```
rclone copyurl "filesender-download-link"  allas:bucket/object
```

Containers

Q1: Are containers executed from the concole?
- Answer: Yes. Actually, one can execute (or run) an image and then it becomes a container. Please check here (https://docs.docker.com/engine/reference/commandline/run/) for more details on different options. For docker you need sudo access rights and for singularity, you can launch a container with normal user rights
Question 2: Singularity for Mac? Could not find a command-line version just a “beta desktop version”
- Answer: Some virtual box was needed, so it wasn't super straight-forward… More detials if interested are here: https://singularity.lbl.gov/archive/docs/v2-3/install-mac
Question 3: So I guess for a Mac user the strategy for building hpc compatible singularity images would be to build using docker (have it installed) and convert to singularity on Puhti?
- Answer: That would work, or then installing a VM with some Linux OS. Note: Ready-made Docker images are sometimes written so that the applications are run as sudo which causes problems in Docker-to-Singularity-conversion, but in most cases this conversion works.
Q4: I have a notable lag inputting text with sinteractive. Is it my connection or a feature of the system?
- Answer: Many people are doing the same thing at the same time now on the interactive node, so this might be causing the lag. So usually the situation is more pleasant. With THESE EXERCISES, it's ok to use the login node (instead of the s interactive queue), as these can be considered light weight tasks, if this lag is terrible.
Q5: I wonder what am I doing wrong. Trying to open the text editor by typing 'nano' in the console but nothing happens…?
- Answer: You need to module load nano first!
Q6: Follow-up Q to Q5: After creating test.sh, the following error appears:[llindfor@r07c52 cont]$ sbatch test.sh
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
- Answer: Check that you have the correct project number determined!
Q7: Where should the output come for the batch job test.sh?
- Answer: There should be a file called slurm-<jobid>.out in the folder where you sent the job.
Q8: When using R in Puhti, we use the command "module load r-env-singularity" to load it. Is there some connection to the singularity with this?
- Answer: Yes, this R has been installed using Singularity!
- There are wrapper scripts using singularity_wrapper for commands R and Rscript. You can take a look at the scripts with:

module load r-env-singularity
less `which R`

Q9: Error writing test.sh: Permission denied. Why's that…?
- Answer: If you can't write a file, you may be under wrong directory where you don't have write permission. If execution is the problem, you can first try chmod u+x test.sh and then try again
Q10: The singularity build pytorch_20.03-py3.sif docker://nvcr.io/nvidia/pytorch:20.03-py3 command takes really long time I’ve been on INFO: Creating SIF file... for almost 10 minutes
- Answer: It's a heavy operation, and it does take time… Maybe we should find a smaller image for the exercise! :) Like this one: singularity build alpine.sif docker://library/alpine:latest
- You can just ctrl+C to kill the process if it is taking long time, we don't really need it for anything, as this is just a tutorial :)
Q11: Is it faster ifs we use more cpus/threads? Like: sinteractive -c 4
- Answer: Yes, multiple cores makes it faster. In other partitions you can get more cores even. Also consider using local fast i/o drives;
Q12: Feedback: I wish that in all tutorials there would be a comment e.g. on the right side of the commands that we are supposed to run in console. Its often difficult to understand what each line is actually doing.
- Answer: Thank you very much for the feedback! We are actively developing the material, so all feedback is highly valued!
Q13: Do we have sudo privilige inside the singularity container to e.g. modify program settings?
- Answer: You have the same user rights inside the container as you have on the host. So if you are running the container as normal user, you also have only normal user rights inside the container. If you run it with sudo, you have sudo rights (not possible on Puhti, of course).
Q14: Can we use the singularity inspect to see what tools are installed inside the container?
- Answer: Sadly no, not really. Open a shell in there, and see what is where.
Q15: Env?
- Answer: env is a command that prints out all the environment variables.
Q16: I'm seeing lots of warnings?
- Answer: Try unset XDG_RUNTIME_DIR -this will hide the warnings :) We could add this to our tutorial!
Q17: Singularity wrapper can be used as a replace to --bind, meaning that it binds the host dir with inside the container itself?
- Answer: Singularity_wrapper does:
  - -B /users:/users
  - -B /projappl:/projappl
  - -B /scratch:/scratch
  - -B /appl/data:/appl/data
Q18: So, outside the sinteractive it would be: export SINGULARITY_CACHEDIR=/scratch/project_200XX for both cache and tmpdir?
- Answer: Yep!
Q19: Is there some good community web-page covering nextflow pipelines combined to containers? I mean something other than nf-core?
- Answer: Check this: https://www.nextflow.io/docs/latest/index.html
Q20: At the uncomment step (// publishDir 'results') I can't seem to find that line in fastqc.nf.
The only "uncommentable" line is // publishDir params.outdir. But params.outdir = "results" has been definied previously, so I think this should work the same wasy, no?
- Answer: Yes, it was defined earlier, so it should work that way.
Q21: How to to define <image_path> in here nextflow run <nextflow_script> -with-singularity <image_path> How do I know which of the files is the image?
- Answer: shub://IARCbioinfo/nf_coverage_demo:v2.3
- Or if you are using a local image file (like we did yesterday), provide a full path for that.
Q22: I didn't know we cannot use "nano" in the sinteractive! Wondering how should I open txt files?
- Answer: You can use nano, you just need to first load the nano module: module load nano
Q23: Can you explain again the thing about over-writting the param file?
- Answer: reads = hardcoded, params.reads = changeable
Q24: cPouta is free for researcher associated with University? And in future it will be good for sensitive data? But if project is over can it be still used e.g. Google cloud or amazon and how much is price (approx). –Do billing units apply in the case of Research in higher education institutions (=one of the free-of-use cases)? –Why do you call them billing units, that's confusing!
- Answer: cPouta, like many other CSC services (Puhti, Mahti, Allas…) are free-of-charge for academic research, education and training in Finnish higher education institutions and in state research institutes. ePouta is the current service for sensitive data. Francesca presented the SD-services that are atm under development on third session, that material is available in e-lena. Regarding the pricing, see: https://research.csc.fi/pricing
  More about terms of use etc: https://research.csc.fi/terms-of-use-and-pricing?inheritRedirect=true
  Billing units are these things we use to measure the usage of our services: if your use case is one of the free-of-charge use cases (like research in higher education institution here), you just apply for BUs, but they don’t cost really anything.
  See the billing units chapter here: https://research.csc.fi/pricing
  We call them billing units, because someone is paying for the use: if not the end-user (like in the free-use-cases), then it's the ministry :)
Q25: How should I change the Singularity cache location to my Project directory
- Answer:
```
export SINGULARITY_CACHEDIR=/some/path
```
Q26:
- Answer:
Q27:
- Answer:
Q28:
- Answer:

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Hint: You can also apply styling from the toolbar at the top

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

of the editing area.

Drag-n-drop image from your file system to the editor to paste it!

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

BioMonth 2021

tags: bioinformatics

Code of conduct

TO DO before the course

Zoom instructions

Image Not Showing Possible Reasons The image file may be corruptedThe server hosting the image is unavailableThe image path is incorrectThe image format is not supported Learn More → Q & A

General and practical matters

Using Puhti

Data management

Using Allas

Containers

tags: `bioinformatics`

Image Not Showing Possible Reasons
The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported
Learn More →

Q & A