# ARCHIVED 2021/01/29 Intro to Computing strategies, Linux,

:::danger
## Infos and important links

* To watch: https://aaltoscicomp.github.io/scip/
* To ask questions and interact (this document): https://hackmd.io/@AaltoSciComp/IntroWinter2021
* *click on the "eye" icon on the top right corner and write at the bottom, above the ending line*
* Questions that require one-to-one chat with our helpers (e.g. an installation issue):
- Finland: ZOOM LINK SENT TO REGISTERED PARTICIPANTS
- Norway: ZOOM LINK SENT TO REGISTERED PARTICIPANTS
* Day 0 program https://scicomp.aalto.fi/training/scip/intro-linux-aalto-computing/
* Day 0 slides: https://hackmd.io/@AaltoSciComp/SciCompIntro
* Triton Kickstart (Mon-Tue) https://scicomp.aalto.fi/training/scip/winter-kickstart/
* If you have just one screen (e.g. a laptop), we recommend arranging your windows like this:

╔═══════════╗ ╔═════════════╗
║   WEB     ║ ║  TERMINAL   ║
║  BROWSER  ║ ║   WINDOW    ║
║  WINDOW   ║ ╚═════════════╝
║   WITH    ║ ╔═════════════╗
║   THE     ║ ║   BROWSER   ║
║  STREAM   ║ ║   W/HACKMD  ║
╚═══════════╝ ╚═════════════╝

---

:::

# 29/01 - Intro to Linux

## Icebreaker - What do you wish to learn today?
- How to look cool like the hackers in movies ;)
- Use more than one screen
- Yep!
- Nested comments. To show
- I would like to learn what new learners are most missing when starting to work in scientific computing
- Learning a bit about Linux would be sweeet : D
- I would like to refresh my linux knowledge
- I would like to learn efficient and right way to connect to and use a computing cluster
-yee!
- Linux 🐧
- An overview of computing resources for researchers
- In Denmark, three Universities (SDU, AU, AAU) operate UCloud a type1 (interactive HPC) resource, which officially opens up for users in march 2021. The goal is to connect all HPC types (interactive, throughput, large memory, accelerated) through UCloud, which includes DTU. read more here: https://www.deic.dk/en/supercomputing/national-hpc-facilities. UCloud offers a GUI with dozens of containers (like JupyterLab, RStudio, Overleaf, Matlab, ...) to spin up on demand and connect to a virtual filesystem , allowing users to connect to network drives, upload and share files with containers and other users.
- What is nird toolkit and how to get access. Working with the data directly where it is stored.
- Not to be so clueless anymore ^^

# Questions
*Ask anything, write at the bottom*
- How is this transition to network based storage (e.g. cloud) influencing the memory of the computer? Or are they completely separate?
- Depends. If you only use the cloud for storage, it acts like a huge hard drive (e.g. Google Drive, Dropbox). However, what is increasingly popular is to use the cloud as a computer (e.g. Google Co-lab, Mybinder, Amazon AWS), so then your own computer doesn't matter anymore, all that matters is the computer in the cloud, memory and all.
- Huh, I see. I would have though that's like a hard drive, like you said, but i didn't think you can use it as an entire computer.
- "The cloud" is basically a buzzword for "the internet".
- Haha good one!
- I used Google Colab once, instead of Jupyter Notebooks, but how is that acting as a computer? Because I used it only to work on code together with other people..?
- When you press "run" in co-lab, the computation is performed by a computer in the cloud, not on your own computer.
- Oh. I didn't know that. Thanks!
- Can we get the link to the lesson?
- It's at the top of this document ^
- I think we still need to find the link to Enrico's page
- You're right. I've added it to the top links now.
- Here: https://hackmd.io/@AaltoSciComp/SciCompIntro

- Is it better to use a generic language such as Perl instead of bash for scripting/automation?
- Perl is not as used as before and many now prefer python but it should not matter that much when starting. However, if you do not know any languages, starting with python for scripting and automation, combined with linux bash is a good option.
- Thanks! I know a little python a bit more Perl. Maybe python is the way to go
- Perl will always have a soft spot in my heart. It was one of the first programming languages I learned. Its sucessor (Perl 6, now Raku) is pretty cool
- It's a matter of taste. For complicated things, I would say it's better to use Perl or nowadays more common: Python. But for simple things, Bash is pretty great.
- Anything that is preinstalled on the machine(s) where the script needs to run should be fine. Doubly so if other people who need to run / maintain the script are familiar with the language :)

- What about the legality of code when using a free online platform for computing? I mean, do you "give away" your code in a sense when using AWS?
- Most of the cloud systems (Colab, AWS) give resources away as an advertisement. Haven't looked at the terms and conditions, but I highly suspect they only record information on who uses their system. Naturally, no proprietary (contract prohibits) or private information should be uploaded to cloud systems.
- As a general rule: don't put code you don't want to "give away" on someone else's computer. However, there are little reasons to keep scientific code secret. One of them is if your code contains privacy sensitive data (volunteer names, other info that needs to stay private).

- How about if the code needs certain libraries to run? In python there is virtual environment concept but for a c/c++ code, some installation will be needed. Is it handled by the admin?
- In Aalto we install commonly used software and help users install their own software, but when dealing with cloud systems you're basically on your own.
- That of course depends! :-) If the libraries are useful for multiple persons, then of course your *friendly* Admins will help. Best if you can test the libraries yourself.
- Admins cannot possibly install all packages you would ever need. Sometimes you need a weird niche package, which you should just install for yourself. The different cloud systems have different ways to do this.
- So the user have the rights to install stuff as well, not just submit jobs to run?
- As the HPC systems are by definition shared systems, one cannot do `sudo apt install` or something like that, but most software does not need admin/root rights to install.
- So the gcc versions etc are handled by admins.
- The users can usually install libraries etc. to their home folder. This is your private disk area. Great!
- For example `pip install --user pandas` will install the pandas package in your home folder.

- I'm working through a VDI (mostly to have access to some programs that need licences to work and the university provides them). I never seem to understand, when I'm connected to the VDI, what internet am I connected to - my own, or the one from uni, which bandwidth matters?
- If you open a browser *in* the VDI desktop, it sees the internet through the Uni connection.
- "your" bandwidth is mainly used to move the image of the desktop to your home-computer and your mouse/keyboard to the Aalto VDI machine.
- So, if the connection to the VDI crashes it's because of my home network, but if something crashes inside the VDI it's the Uni network.
- Yes!
- Great, thanks!
- As a more security-inclined question: when you're working inside the VDI, how secure is the connection? I'm specifically thinking of the security you have installed on your own machine, but on the virtual desktop there are other securities installed - do they not crash or get overwritten?
- I don't know the technicalities, but "standard security" is applied.
- By "security", do you mean anti-virus or what exactly?
- Ah, not the actual anti-virus, but more your data on the internet. I always have a feeling, when I'm on the VDI, that anyone can access my data.
- Your data is accessable only with your username/password as with any computer.
- The access to the VDI is limited to the browser / vmware horizon client you're using and the graphical information / key presses are encrypted when they are transferred between your client and the VDI server.
- Nice, that's what I've been wondering!! Didn't know exactly how to put it into words, sorry! Thank you for figuring out what I meant :)
- It is a very valid question. Some other remote desktop applications like vnc servers / clients can sometimes be hard to completely secure. VDI is much more secure than such tools.
- VDI provides Equivalent safety as going to the physical workstation. Different threats, sure
- Always threads on the internet! The best we can do is minimize them, huh.

- The number of VDI machines with "nvidia" -- a graphics card -- is limited. I often run out of them.
- If you're doing something like machine learning or deep learning on GPUs and running long training sessions, we highly recommend using the non-interactive runs on Triton as the GPUs there are much more numerous and powerful. Of course, if you're doing data analysis e.g. with rapids or testing that your training code works, VDI is a good option.
- Try also Panikki for more machines with GPUs https://scicomp.aalto.fi/aalto/paniikki/

- I sometimes ssh into Maari computers with names such as sitriini, zirkoni, rubiini etc. Which part of the Aalto structure do those computers belong?
- They are normal workstations, so they are similar to the VDI machines, just physical instead of virtual. Of course they might have other users on there at the same time so running e.g. unsecured jupyter notebooks (no password) might give these users access to your account. Think of it as a shared computer. If you lock your session, somebody else can log in with their account and if you leave something accessible to them (e.g. data in /tmp, open services like jupyterhub), they can access them.
- They are run by Aalto IT Services, by a similar team that does employee workstations (but they are targeted to students)

- Is Paniikki! only for CS students?
- Everyone! (at Aalto)

### UiO specific questions
- What is the security level on jupyterhub?
- Up to Green and Yellow data. For clarifications please contact [e-mail]
- What’s the best way (wrt to stop using the resources) to log out from a desktop? “Disconnect” or “Sig out”?
- Logout or signout is enough (shutdown not needed)
- If you are working with pseudoanonymized patient data . Where is the best place to save the key that links to patient names, IDs? TSD?
- Black level
- See https://www.uio.no/english/services/it/security/lsis/data-classes.html for data classifications - deanonymizing medical data is at least at level red, probably black
- On paper. In a binder. In a locked cabinet.
- Support for Data management when writing NRC applications
- Take contact to [e-mail]
- Also take a look at
- https://www.uio.no/english/for-employees/support/research/research-data-management/index.html
- https://www.uio.no/for-ansatte/arbeidsstotte/fa/ (English link at the right-top corner)

- I use vdi to access windows. I have seen that the computer gets formatted quite frequently. Why is it so?
- Do you mean that some programs go missing?
- Each time you get to a different machine, so only your "home" files stay the same... and they are not real machines (virtual machines)
- For example, I open a word file, leave it open and log out, next day I see everything is closed. The download folder also gets cleared.
- The virtual machine will expire after a certain time it's been unused. You should save all intermediate work done to home folder. As mentioned previously, the next virtual machine starts from an empty state.
- Also if you log out, VDI will close all running programs and remove the virtual machine.

**We are back with... LINUX SHELL!**

* If you have just one screen (e.g. a laptop), we recommend arranging your windows like this:

╔═══════════╗ ╔═════════════╗
║   WEB     ║ ║  TERMINAL   ║
║  BROWSER  ║ ║   WINDOW    ║
║  WINDOW   ║ ╚═════════════╝
║   WITH    ║ ╔═════════════╗
║   THE     ║ ║   BROWSER   ║
║  STREAM   ║ ║   W/HACKMD  ║
╚═══════════╝ ╚═════════════╝

# Questions here below
- Do we need to install python? I get command not found.
- I don't think you need it for this part.
- Thanks!
- it is good to type along the same commands!
- You can, but do note that e.g. folder `test` is most likely not present on your system. But running `ls` to see how it works might be a good idea.

- Why do some options have a single dash (-) and some two dashes (--)?
- Typically single dash `-` options are for single letter arguments while double-dash `--` options are for multi-letter arguments. For example `ls -a` and `ls --all` are the same option, just written differently. `-a` is just an abbreviation. If the option takes an argument one typically writes it as `-o argument` for single dash option and `--option=argument` for double-dash-option. For example `ls -w 10` and `ls --width=10` are the same option.
- Tnx!
- Are there major differences between less and more? I use more often.
- `less` lets you scroll back up again. `more` only scrolls down.
- From `less` manual page: ``less - opposite of more``. Less gives you less output (you can scroll back), more gives you more output.
- I think in many modern linux distributions, `more` is linked to `less`, so there is no difference any longer.
- less can do many tricks vi can do, e.g. search "/"

- Can I copy several files at the same time?
- Yes! `cp file1 file2 file3 file4 output_dir/`
- This copies 4 files to the folder `output_dir`
- Often one uses wildcards (comes later in the material most likely). For example `cp file* output_dir` (copy all files that start with `file_` to `output_dir`)
- The general rule is that you first give all the files you want to copy and finally put the destination at the end. When copying multiple files, the destination needs to be the name of a folder (directory)

- Why is it a good idea to use Ubuntu (or Linux in general) when working in research/as a student? Better than Windows for example. Does it come with specific advantages?
- Opinnion based...but I believe Unix/linux is designed as a tool for professionals while Win/Mac is for "normal" end users. This explaines why power-using linux is more natural than power-using a windows computer.
- All supercomputers are linux based.
- Lots of research software is developed by researchers that use Linux for developing software as installing libraries & compiling code is usually much easier on Linux.
- If you want to try out Linux on windows machine Windows Subsystem for Linux is very useful. It basically runs an Ubuntu installation on your Windows machine. These kinds of features were created by Microsoft to entice programmers to keep using Windows and they are very useful.
- It is a good idea to learn Linux because for your research you will probably need to scale your codes on larger machines and most of the time these machines are Linux based. However, for your laptop Windows, Mac-OSX or Linux are not very different (because you will be using remote access).
- Once you know a little UNIX it is easy to automated things and make your own baby-software called scripts
- Okay, I get it. I wanted to know the advantage - which I now understand that it's for research when you need to scale the codes, but on day-to-day basis it's not different. That's how I would get experience and end up having an opinion :P I can't if I don't have a reason to learn to use it in the first place, right?
- Depends on your field, but if you're working on physics most codes are almost impossible to compile on Windows as they use open source mathematical libraries & MPI libaries. On ML/deep learning fields some GPU codes only work correctly on Linux as they need low level access to the GPU hardware. In statistics, R requires lots of libraries so that it can install packages. Most of these problems can be solved in Windows, but they are still problems. So one needs to make extra work. This usually drives people to use Linux.
- You use what is most efficient for you, but it is convenient to have the same environment and modifications to your environment in both your desktop and at the "target" system. Many times your workstation is a very good and representative example of a larger system.
- I see, it's about perspective. Something to think about, really. Right now I'm not working much in Python (I used to, and didn't learn Linux), so maybe next time when I need to run these type of codes I'll consider it. For most softwares like R or Qualsis or even VSCode, Windows works fine, but I understand than when I'll need to get to some low level access, Linux will probably be the go-to. Thanks!
- I have the *feeling* that linux is more flexible and adjustable, but that is probably only due to knowing less on how to do the things on Windows.
- Please, make sure you do not give your opinion but an answer that is correct. (Even when there is no correct answer.)
- That's a good point, but I was looking for an opinion as much as for a "correct" answer regarding the Linux advantages, so this all helped! :D
- Personally, my own computer is in Windows, as I want to play video games on it. If you have ever tried to run PC games on Linux, you know how hard it can be. Conversely, I can do hardly any serious work on it without taking a connection to linux system.
- Quick note on Linux gaming (which hopefully won't crash your productivity): it's become _way_ easier in the last couple of years thanks to Proton and Vulkan. Nowadays, you shouldn't run into problems running _most_---loosely judging by [Proton.db stats](https://www.protondb.com/)---Windows games on Linux (unless it uses some of the anti-cheat software which mistakenly classifies Linux as an "unauthorized modded Windows").
- Or simply use Microsoft office programs

- What is the PATH
- PATH is an "environment variable" that the shell uses to look for programs. All folders in the PATH variable are searched for programs when you try to run them. E.g. when you run `ls`, the shell searches from the folders in PATH for a program called `ls` and when it finds it, it executes it.
- Good question! I will ask RIchard to comment on it

- How can you get cmds logged to _bashhist? Sounds useful
- they are logged to your home folder .bash_history automatically and with the command "history" you can see them
- You can also have your current history mapped to a file like it's done here. I have a command for that but need to find it first ^_^ (I just use `history`)
- Yeah thats what I was interested in :) Log in folder. Sadly terminal doesnt do this yet so was gonna have my ubunto-session do it itself.
- Many options listed here https://github.com/coderefinery/manuals/blob/master/instructor-tech-setup.md#terminal-history-for-learners
- Ty!

- Could you also talk about soft links in linux?
- "soft links" are files that contain the path to another file. (In windows, these are called "shortcuts"). Whenever Linux is asked to open the file that is a soft link, it will open the file indicated inside the soft link file instead. You can use this feature to create "fake copies" of files: the file appears in two different locations, but is backed up by a single file. So any edits made to the file will be visible from both locations.

- Advanced bash scripting guide
- Linux shell tutorial:
- https://scicomp.aalto.fi/training/linux-shell-tutorial/
- Aalto Scientific Computing shell courses:
- https://scicomp.aalto.fi/training/scip/linux-shell-basics/
- https://scicomp.aalto.fi/training/scip/shell-scripting/
- .
- .
- .

- Can You please give an example of bash file that applies 10 cpus + gpu #1 to run a program so-called sample_code.run?
- We get to this on monday and tuesday as part of the Triton cluster kickstart

# FEEDBACK FOR THE DAY
*Please let us know one thing that went well and one that could be improved (it's the first time!)*
- .
- interactive question/answer section was very useful.
- It will be beneficial if we can see examples of commands needed to submit jobs on the cluster Thanks!
- Indeed! This is exactly what we will cover next week
- What went well: the vertical screen and having this interactive page for questions was new to me (and I was a bit skeptical in the beginning), but I think it worked great! Super good idea!
- To improve: it might have been only me, but there were a loot of different pages and links to look for. This might have happened because of all the platforms this workshop used, but just saying that I needed around 15 minutes before it started, to have everything ready. (Didn't mind that much though.) You should still keep the twitch stream and this hackmd for questions/discussions, though!
- +1
-

----
:::info
^^^ Please write above this line ^^^
:::