# Linux Shell Basics / August 2023 (Archive) :::danger ## Infos and important links - To watch: *link will be sent* - Previous of this document are archived at: https://notes.coderefinery.org/shellbasics2023archive - Program and materials: https://aaltoscicomp.github.io/linux-shell/ - Presemo link: https://presemo.aalto.fi/shell - Lecture's demospace copy: https://users.aalto.fi/~degtyai1/shell/ - Prerequisites: - Make sure you have a working Linux terminal on your machine or access to a remote machine running Linux. - Suggestion: if you have just one screen (e.g. a laptop), we recommend arranging your windows like this: ```bash ╔═══════════╗ ╔═════════════╗ ║ ║ ║ SHELL ║ ║ ZOOM ║ ║ WINDOW ║ ║ WINDOW ║ ╚═════════════╝ ║ WITH ║ ╔═════════════╗ ║ COURSE ║ ║ BROWSER ║ ║ STREAM ║ ║ W/Q&A ║ ╚═══════════╝ ╚═════════════╝ ``` - Linux Command Line Cheat Sheet https://cheatography.com/davechild/cheat-sheets/linux-command-line/ * **Do not put names or identifying information on this page** ::: *Please do not edit above this* # Icebreaker **Test how to use this hedgedoc live document, this is how we ask questions and get help.** - This is a question, or is it? - This is a reply, yes it is! - This is a nested comment to the reply - Actually... insert smart comment here - This is a different reply, but still valid - This is another question, or maybe not? - ... - ... - test it. - ... write something... - just testing... - more testing! - Hi,everyone, do anyone know how to d - ... d...? delete? download? :) ### Share a story of when you had to do the same (research) thing over and over again: - My advisor came to our offices and asked each person to do several parameters of a simulation and email the results back. - That's parallelization! :) - Good afternoon. How will we be able to get access to the recordings of these trainings? - We'll post to youtube and link from website as soon as we can. - Alright. Kiitos! - - Hello, How are we supposed to open a terminal window as a windows user without an aalto acount? - Are you at univ of helsinki? - yes! - start powershell and then type ssh USERNAME@SERVERNAME where servername can be login.physics.helsinki.fi - What is a powershell please? - A software that comes with windows (the windows own terminal, which is not compatible with Linux) - Alternatively you can install "gitbash" a program that simulates the Linux shell for windows. - or you can run the "simple" windows shell (Start -> Run -> type cmd) ## Basic shell https://aaltoscicomp.github.io/linux-shell/the-shell/ - This link shows different ways to get a shell which you can use for this course - If you are not at Alato University, check different ways to find a Linux server to connect to. There might be ideas on your organization's web page - . - . - Hei there! Is there a difference to say echo 'hello' or echo hello? would it be a mistake? - There is no difference in the output. The difference is that by using '' you are indicating to the echo command that the contained data is to be interpreted in a specific way (i.e. in this case as a literal string). try: `echo 'hello $HERE' there $HERE` and see what happens with the two $HERE - we'll see more about quoting later, good question! - . - . - Can we consider Bash a programming language? - yes! A progarmming language specialized in managing other programs, not for typical programming language purposes. ## Starting out https://aaltoscicomp.github.io/linux-shell/starting-commands/ - Why does some folders have '.' as the first character? - These are "hidden directories". There's actually nothing special about the actual name, some commands simply don't list them by default (like the `ls` command). - Pasting into the terminal adds random symbols, what's the best way to paste? - Hm, that's a hard one. What terminal and terminal program are you using? (VDI with web browser?) - I'm on a basic Linux shell. Hints? - See below (ctrl+shift+v) - I'm on University of Helsinki Cubbli VDI, opened a Terminal - Check if CTRL+SHIFT+V is a better option. But sometimes VDI has a strange behaviour with copy/pasting. If you are pasting from the pages of our course, you can open those pages in the same VDI session so that you know that your copy/paste stays inside the VDI. - Ctrl + Shift + V worked, thank you! - In general different things to try: control-v, control-shift-v, right click and paste, middle click. Depending on what terminal app you are using but if problems can't hurt in trying them all. - What is the difference between ls and dir?. - Two different programs that do similar things. Slightly different output defaults. - ... and in modern linux, basically the same. `ls` is the stanard linux one, `dir` is an alias with some default options. - What is the difference between the terms: unix and linux, if any? - short answer: 'UNIX' is a general term for a broad type of operating system (started in 1970s). MacOS is unix, too. "Linux" is the common term for desktop that's common now (Linux kernel + GNU tools + all the other common open-source stuff made in the last years.) - long answer: technically "linux" is only the very low-level kernel of the desktop. There's far more than makes it possible than that! But it's become the common language name. - Clear, thanks! :) - Comment: sometimes `command --help` is a good alternative (e.g. if `man` is not installed) but it depends on the command +1 - The 'pwd' command outputs 'u/11/user/unix', what does the 11 represent? - At Aalto University, that `NN` is basically random and used to divide home directories up to different storage systems - but surprise! it's also the last two digits of your user ID (run `id`), not that it matters anyhow. - Is it possible to use emacs while working on Triton? I meant graphically - Yes, it should be installed. (do you mean use from terminal or graphically?) - Anyone have a recommendation for a good shell cheatsheet? (optimized for scientist and researcher type people?) - how to 'unclick' in presemo? - ~~Clicking again works for me~~. OK, not for this question. Sorry. - I have a silly question, what exactly ssh is ? I heared about the the putty how it is different from ssh - `ssh` is a program to connect to other computers: it only gives the shell, so it's small, efficient, and secure. It basically makes one terminal connect to another computer. It's really one of the things that makes all infrasturture in the world work! (ssh lets a lot of other stuff work too, like network connections and file access.) - Putty is one ssh program. It also includes a terminal on Windows. - I see, thanks - Why there is `#!/bin/bash` in the first line - That's the program used to run the script. This is a bash script, so it should be run with bash. - So, just as an example, you could also have `#!/usr/bin/python` and write the script in Python. - do does it mean that bash can execute not just the bash but also python script? - Inside details: `#!` actually tells the operating system "use this program to run this script" (run `PROGRAM SCRIPT_NAME`). So it's not bash running python, but the operating system directly going to Python (or bash, or whatever). - This goes a lot beyond only "bash" and "the whole unix environment", but they all go together to make this cool thing that can do anything! - What is the difference between making a script executable with chmod +x <file_name> and inputting command bash <file_name>? I've always been running scripts with bash <file_name> and never used chmod +x ... - `#!program_name` is the technical equivalent of `program_name script_file_name`! Choose whichever makes sense. I usually use `program_name script_file_name`. - Why is Ivan able to run 'help.sh' without using 'bash help.sh' or 'sh help.sh'? - So first is `chmod a+x help.sh` which makes it executable. Second, `.` (the current directory) may be in the `PATH` environment variable which tells where to look for programs to run (so that `help.sh` would find `help.sh` in the current directory). We should ask when Ivan comes back this is a good qustion! - How can %CPU exceed 100%? - can any body shre the link to the question? - https://presemo.aalto.fi/shell - I lost it ---Could you please tell me again how to swtich between the screen, I did not get it? ## Presemo https://presemo.aalto.fi/shell Could you please go quickly over the correct options? * FALSE BASH can only execute built-in commands * TRUE BASH, as any shell, combines other programs to make it to work together * TRUE BASH documentation can be found on the web * TRUE man pages is the way to get info about commands * FALSE man pages available for the external programs only, for the built-in one has to use 'help' * FALSE type -a' tells about file content's type * TRUE BASH is also a scripting language Thanks a lot ## BREAK :::info ### Break until xx:10. Friendly reminder: next course in 3 weeks -> https://coderefinery.org/ ::: ## Processes https://aaltoscicomp.github.io/linux-shell/processes/ - `top` on Mac looks a little bit different than standard Linux. Windows implementations (like gitbash) do not usually have a `top` command. - Is there a way to be sure we are not creating a big mess while killing a process? - You shouldn't be able to kill anyone else's process. So no worry about messing up someone else - For yourself: don't kill anything you don't know of. Things like bash, Python, stuff you have started yourself is usually OK. It'll end without saving but is usually OK. - What about if my process (which I have started) doesn't respond to killing? Is it a good idea to kill its parent? - You'd usually try `kill -9` and see if that works. You could try the parent. Sometimes stuff can get really stuck at a low level, and nothing works. Let's just hope you don't get here!, but usually I'll ignore it, make new shell and wait for reboot. - If this happens I would be 100% sure I have messed up xD - A common case is computers with network file systems: filesystem doesn't respond, program hangs, doesn't respond to kill. Just assume it's something with the operating system. And that you can ignore and go on to other things. https://presemo.aalto.fi/shell - It is not important, but I am just curious. Who can kill zombie processes? - I am not 100% sure but I think zombie processes are already "dead" so cannot be killed. If the parent can be killed then they disappear. - Mostly system reboot fixes the issue. Then, as a sysadmin, you have to find out what was the reason for the zombie processes. - I reset my shell, and still have a background job called '[1]+ Stopped grep --color=auto clear' Why's that? - reset how, with `reset` ? - yes! - that basically re-syncs some the display (terminal size, etc), not what's actually running. Best way to get a real reset is to get a new shell. - great, thanks. - What's the meaning of the + and - next to number? - good question, I never really thought of that. I found this: https://unix.stackexchange.com/questions/186759/minus-and-plus-sign-in-jobs-process - `+` means default for `fg` command. `-` means next-up for `+` if the `+` job ends. - yes, if you run 'fg' with no job number, you get the one with '+' - WHats the difference between killing via process id or via jobs? - Hm, good philosophical question. I think basically similar. `jobs` only shows jobs which this one very shell knows about and has a different id for convenient use in that shell. `kill` is generic. - How to kill screens? - easy: exit all the shells in it - more advanced: "control-a control-k" until all screen tabs are gone. (I'm actually not sure what I'd do to kill the whole thing!) - sorry this screens part was a bit confusing.. how do I know what screens are active and manage them? thanks - https://devhints.io/screen seems like a reasonable cheatsheet. Most common I use: - `screen -ls`: list screens - `screen -r [name]` reattache - `C-a C-d`: detach screen (control-a, etc) - `screen -XS <screen.ID> quit` to kill a screen from outside - How to avoid 'screen within a screen'? E.g. use 'screen' in a shell on my local computer, then 'ssh user@triton.fi', then 'screen' again on triton? is it meant to be used like this? - I follow because I don't know the answer myself :) however when I work at the actual local computer, I do not use screen. I.e. screen is only run on remote computers (i.e. only one screen instance is running). - thanks! I am confused about how to use screen to 'stay connected' - First you connect to remote server (e.g. ssh triton.aalto.fi) and then you start screen (or tmux) the first time. Then you will disconnect at some point. When you connect again you just resume the running screen (or tmux) and everything will be still there (unless the remote server was rebooted of course...) - Cool! Thanks, will practice this! - Thanks for the tip, very useful! ## Exercise :::info ### Exercise until xx:50 Exercise: https://aaltoscicomp.github.io/linux-shell/processes/#exercises-1-1 Mark your exercise progress at: https://presemo.aalto.fi/shell ::: - What does "which" shell mean? :D - options like `bash`, `zsh`, (probably one of these), `tcsh`, `csh`. - hint: `echo $SHELL` - Thanks - Will there be answers for this excersise in the end? - yes! - What's a hot key? - Keyboard shortcut basically - Will the material be available for participants after the course if at some point need some "memory refreshment"? - It's all open source and should remain where it is (with improvements) as long as Github lets it stay! Send improvements. - Why did I get the error "no matching criteria specified" when using pgrep -l? - what was the whole command line? - oh no. It's "no matching crietiria specified", right? You need to give some pattern of which it matches. - - Can I use search command in man? - In the man page that you have opened you can type /STRING (replace string with what you want to search for). It is basically the same as doing a search with terminal editor "vim". - I am using the VMware remote server from a Mac. When I am using nano, I cannot sent the process into the backgroun with "bg". It will write the g into the file. Is there a different combination of kezs for Mac? - on my computer (linux), C-z doesn't work with Nano... some programs can prevent it from working. Does C-t C-z work (control-t, release, then control-z) ? - Yes, that actually works. But is the process then killed or just in background? - In the background - Where the recording of the lecture will be available after the course? - We'll try to process it quickly and release ASAP. Hopefully it's fast if there's not much personal data in it. - Where will the links be? - We'll at least add it to the course page. And probably it'll go out it some email. It'll be in this channel: https://www.youtube.com/@aaltoscientificcomputing3454 - .What does | do? - It redirects the output (STDOUT) of the command on the left into the command on the right. For example ` ls | grep a` lists all the files with `ls` and passes the output to command `grep` to filter only those with the letter `a` in it. - . - How can we find the pid of a backgroup process? - just answered (`jobs -l`) ## Feedback Please check presemo for quick survey (21 fine, 9 too fast/complex). If you have more detailed comments on what to improve, please write them here. Write also what went well. :) - I was able to follow but the content is slightly challenging. More time for the exercises could help. - ... - ... - ... # Day 2 - The presemo: https://presemo.aalto.fi/shell/ - ...please share the presmo link? - see the line above - I see ## Files and directories https://aaltoscicomp.github.io/linux-shell/files-and-directories/ - could you show example cases with each of them? Okay thanks! +1 - Ivan will give some examples, however this is examined with more details in part 2 next January. The material that will be used in part2 is in the same site https://aaltoscicomp.github.io/linux-shell/quoting-substitution-aliases/ - How can I open the window with my cmd history in the same way as you did? - See the "instructor's guide" on the same materials https://aaltoscicomp.github.io/linux-shell/guide/ (so you can have the same setup as if you were the instructor) - Are there wildcards to discriminate capital and lower characters? - Not directly. This seems to work for me for example: `ls [a-z]*` vs `ls [A-Z]*`. But that's really just giving some limit and may not be useful for what you need... - just a comment that I find using eg. FileZilla handy when playing with files and directories - Once again, what was rm file 1?.txt supposed to do? - `rm` deletes files... `1?.txt` expands to "anything with 1 and .txt with one character in the middle". So the shell finds all files that match, passes the names to `rm`, and `rm` removes them - if you are not sure: before deleting many files at once I like to run "rm -i 1?.txt" which means "interactive" that will ask me "are you sure you want to delete file 1a.txt?" etc for each file - +1 - If you get `du: .[!.]*: No such file or directory` it means that you do not have any hidden file in the folder where you are running the command `du -sh * .[!.]*` - +1 - What are lrwx and drwx? - those starting with "l" are links, not real files but links to a file. We skipped this part. You might be familiar with links even with graphical interfaces where you might have icons on the desktop that open a certain folder or certain programs (which are not actually stored inside the Desktop subfolder) - those starting with "d" are directories - would it nice to make the session slightly slower +1 - If you are a mac user, when you run `ls -l` you might get some files with permissions `-r--r--r@` the `@` is "extended attributes", extra metadata that mac uses for the file - the 4,2,1 (r,w,x) come from the binary 100 010 001, so the binary 111 has the switch on for r w x i.e. the decimal 7 - `-v` stands for what? - makes chmod verbose, it tells what's going on. - `man chmod` then inside man type `/-v` and it searches within the man page the meaning of the option `-v` - Thanks - `-v` often makes output of unix commands more verbose, so if a command fails, it often helps to check the verbose output - - what tryton means here??? - It was probably Triton, Aalto's computing cluster - is it possible to get the this recording to watch again???? :::success ## Exercise 1.2 (until xx:05) - https://aaltoscicomp.github.io/linux-shell/files-and-directories/#exercise-1-2 - Mark your progress on presemo https://presemo.aalto.fi/shell/ ::: - `/usr/bin/[` is an actual bash built-in for testing. It is the equivalent of command "test". It tests for a statement if it true or false. ## Find https://aaltoscicomp.github.io/linux-shell/find/ - touch -R ~/* # will this touch everything in home folder including subfolders? - I am not sure. I don't see the -R option for touch - thanks. then what would be the right way? - one way of doing it would be using the command `find` with the option `-exec` which means "for each file you find, execute this command". - got it. thanks! - `find . -type f -exec touch {} \; -print` - . - .. - ... - .. - . # Feedback about today If you have any comments regarding today just let us know here below (what can be improved and also what went well) - ... - ... - ... - One serious problem is that the university of Helsinki server Cubbli Linux does not let me type special characters in terminal. Thus, doing the exercises in real-time is a problem. Another limitation is the notebook screen size, with three windows open legibility becomes a problem for the terminal and demospace windows. Just FYI ..., not that I believe you guys can do much about these unfortunate limitations, but it may be good to know about these limitations anyway. # Day 3 ## File archiving and transferring https://aaltoscicomp.github.io/linux-shell/file-archive-transfer/ - Whats the difference between all the compression methods, tar, bzip? - Difference performance. Some are able to compress a bit more than others, some allow encryption. I personally just go with the standard gzip which works on all the OSs. - Can tar also be used for compressing single files? - The command won't throw an error, but then you could just type `gzip filename.extension` and you will get a file called `filename.extension.gz` - how do I transfer from non-linux to linux? - Probably still ssh. "sftp client" is the key word - WinSCP for example. For mac, one can use native ssh client or Cyberduck for GUI :::success ## Exercise 1.3 until xx:55 https://aaltoscicomp.github.io/linux-shell/file-archive-transfer/#exercise-1-3 ::: - . How can I search for a word in a man page? - /seachrword (= slash + searchword) and then 'n' and 'N' for navigating over search results - I am getting 'file is the archive' error when tar. What am I doing wrong? - What is the command you are typing? - tar -cf path/file.txt ~ - are you trying to compress the whole home folder? Also the notation would be - nope.. I was thinking the sequence is (wrong) tar -cf file-to-compress folder-where-to-put - here the correct sequence: `tar -caf arhive_name.tar.gz directory_to_be_archived/ ` - So first you specify where the archive goes and then the files you want to archive. See beginning of page https://aaltoscicomp.github.io/linux-shell/file-archive-transfer/#file-archiving - thanks for clarification, my bad - Does "find -type f -perm /666" do the same thing? - I am not 100% sure but my feeling is yes. It would be strange to find a permission 646 (read and write by you, everyone, but not by the group) - Extract only one particular file to a subdirectory from the archive - tar -xaf demospace.tar.gz -C folder1/ demospace/hello.sh - comment: the order does matter! -C is applicable only to the options that follow after it ## Command line utilities https://aaltoscicomp.github.io/linux-shell/cli_utiltities/ ## Evaluations, separators and grep https://aaltoscicomp.github.io/linux-shell/pipelines-grep/ ## Feedback for the course Thank you for joining us! Leave some feedback on the course :) Anything that comes to your mind: things to improve, things who worked well, future ideas, suggestions... - Perhaps the live session could be outside the lunchtime, since 12:00 to 13:00 is the usual lunch hour. Great sessions, thanks! - A bit more breaks would be good. So much information that impossible to focus this long. Also, maybe less lecturing and more exercises or somehow that we could simultaneously try the commands. - Thanks a lot for the course! May be a bit more time for exercises would be useful. A lot of useful theory, but more practice would still improve the learning. - Thanks guys! Excellent work! - Thanks! Good learning experience! - Very informative and although a little fast-paced im sure the recorded videos will be very useful. Finally, join us for CodeRefinery workshop in about 3 weeks to learn about git version control + reproducible science + good coding practice + research software... https://coderefinery.org/ And Enrico will email you about videos when they are uploaded + future courses to continue your learning path. --- Please always write new questions at the end of the document, above this line.
Last changed by  
 
AaltoSciComp
126

Read more

Triton v3 communication

about 20% of current Triton resources (cpu, gpu, etc). Rest will be migrated shortly.

4/16/2024
FCCI Tech

Quick look at the code: Why should it be published separately?

1/26/2024
Linux Shell Scripting / January 2024 (live)

To watch: link will be sent to registered participants

1/18/2024
Kubernetes application demo

The goal of this lesson is to have a crash course of kubernetes, so that someone (who knows a bit about kubernetes already) can deploy a simple app to k8s cluster. We probably won’t say everything that someone needs, but hopefully you will know where to start reading to accomplish your need.

11/18/2023
Read more from AaltoSciComp
Published on HackMD