Linux Shell Basics, Sept 2020 ================== * Course page https://scicomp.aalto.fi/training/scip/linux-shell-basics/ * Zoom link https://aalto.zoom.us/j/947220751 * Course material https://scicomp.aalto.fi/training/linux-shell-tutorial/ * The HackMD session notes are in the write-mode, feel free to post your questions, ideas, doubts over here, we'll follow them and answer the best we can. For the [HackMD syntax/help](https://hackmd.io/c/codimd-documentation/%2F%40codimd%2Funderstanding-your-editor). Mon session (7.9, 12-15) ------------------------ <details> <summary>Click to expand: </summary> https://scicomp.aalto.fi/training/linux-shell-tutorial/#part-1-linux-shell-basics Icebreaker: what kind of tools does your research use (computing or otherwise) * R and Stan * MATLAB * Pytorch * Python, mainly networkx and graph-tool libraries * Python (numpy and sklearn libraries) and fortran > [] * ... Questions: * Ask questions like this * You'll get an answer like this * Is it OK to use virtual Linux? * Yes, basically for this course all you need is to access a bash shell. * How can we have a virtual linux on our windows? * There are different options, these days [Windows subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10) is a common option and easiest. There are also virtual machines (like virtualbox) or possibly Docker. It depends on what your purpose is. * vdi.aalto.fi is an option for the Ubuntu and/or Windows 10 environments, login and try (requires Aalto account, but other universities may have similar) * If at Aalto, come to our garage and chat: <https://scicomp.aalto.fi/news/garage/> * Are the shell commands in linux case-sensitive? * In general yes, almost everything in Linux is case-sensitive. * Can the session be recorded? * We generally want to preserve privacy, but realize the value of recording. We've tried recording in the past, but the post-processing effort meant we ended up not releasing it. * We'll discuss here again during the break * What is the logic behind the moving processes? * Can you clarify? You mean up and down the list? * top/htop sorts by CPU usage (or other things), as the instantaneous CPU usage changes the order changes in real time. * This process management stuff is quite important to using any system efficiently. * I am trying to access the triton server from the terminal using ssh from my personal laptop. The password is invalid, it should be my aalto password? Is a VPN required? EDIT: The user name might be wrong, since it says "Permission denied" * If it says "invalid password", it is probably able to connect so it isn't VPN (but in general, VPN is required from the public internet) * If password is typed wrong three times (or wrong username), you get blocked for 15 minutes * Is your username correct? Wrong username will cause this effect, too. * We can check more during a break. * If I would run one script, would that be deemed as one process that could be found by "top" -command? Or are these processes always large softwares running (like browser, spotify, etc)? * Everything, every shell command, etc. is a process * If you run a shell script, it will often start other process from it (for example, grep, ls, etc). This is normal. * I am running W10, installed PuTTY, did the SSH connection, etc. Everything works for a few minutes, then I get a fatal error in PuTTY ("Remote side unexpectedly closed network connection"). Could it be due to running a VPN connection or is it something else? * If an Aalto user, try vdi.alto.fi, and run Ubuntu session from there. There is no need for VPN in case you are a vdi user. Otherwise, yes, vpn can be overloaded easily. One could try vpn2.aalto.fi instead. * On my home ubuntu kill PID did not kill the process, but I had to write kill -9 PID, while just kill PID did work on kosh. Why? * Dependes very much on the process you try to kill and its state. Simple 'kill PID' is a nice way of sayong to the process, please clean up after yourself and kill all your childs. If process dores not react, by some reason, 'kill -9 PID' will just kill it and its childs without being nice. * Whats the difference between 'ls /' and 'ls'? * you meant probably ls 'file name'? Single quotes make the the command (=BASH) to work with the input as a single word. ## Exercises * Bottom of this section: https://scicomp.aalto.fi/training/linux-shell-tutorial/#exit-the-shell-and-screen-utility * Is triton down? I get "Resource temporarily unavailable" when ssh triton.aalto.fi * Interesting, it should be up. From where are you sshing? * Remotely on my mac * Do you have a VPN on? * Nope, do I need that? I have ssh without vpn previously. Oh so should I first connect to kosh and then triton? * Yes, normally you do kosh -> triton if not on a VPN. * How can I access to LUT linux service? * I [name=rkdarst] can't find a link yet, asking for more help * You may try https://uni.lut.fi/lut-etatyopoyta, not excatly an interactive server but a virtual environment, hopåefully has Ubuntu or other Linux </details> Wed session (9.9, 12-15) ------------------------ <details> <summary>Click to expand: </summary> https://scicomp.aalto.fi/training/linux-shell-tutorial/#part-1-linux-shell-basics see Files and directories Questions: * Ask questions like this * You'll get an answer like this * What is the name of these bash "operators" such as * in ls /bin/*. And I am asking more generally. The word I was looking for was "glob". Thanks, I needed a search term for google. * Are you asking about `*` specifically or some general concept of operators? * `*` specifically is a wildcard glob: it expands to all files that match that. For exmaple `ls /bin/z*` lists everything that starts with `z` and is internally expanded to `ls /bin/zcat /bin/zdiff /bin/zfgrep ...` before running. * More generally, shell processing happens in several steps: 1) handle quoting on command line. 2) expand different operations, such as globbing math `$$(1+n)`, etc. Then it runs the command. (this is a simplification). The general idea of "what the shell does before running" vs "what the raw command is actually run?" is very important. * I have made a testdirectory, then change the permission, but when again use ls -l to see the permission nothing has changed, whereas by using getfacl it shows the new permissions!? * ls -l testdir * getfacl testdir * What are the two commands you are running? Note that `ls -l testdirectory/` will list the contents of the directory, no the permissions of the directory itself. `ls -ld testdirectory/` will show the directory itself, not the contents. * Yes, try `ls -ld testdir`. You can also experiment with `ls -la testdir` and `ls -la` (no argument) to see the directory permissions. Notes ------ du -hs * .[!.]* | sort -hr </details> Fri session (11.9, 12-15) ------------------------ * I tried to copy a file from my local hardrive to kosh using the command `scp -r my_dir kosh.aalto.fi:my_home`. But at the moment of ssh it requires a password with the wrong username. Can I modify scp with -o so I can add my username on kosh to the command? (Similar to ssh -l username) * try `scp -r my_dir username@kosh.aalto.fi:my_home` (`username@` added) - this is a standard ssh syntax. does it work? * It works, thanks. I tried the `man scp` method ... [[user@]kosh.aalto.fi:]my_dir. But it did not work. * Ah, the brackets mean "what is inside is optional", not to be included * When I type this line, the output is no such file or directory. How to fix it please? jiangw3@kosh:~$ scp -r science-it triton.aalto.fi:/scratch/work/jiangw3 * Does the`science-it` directory exist on kosh? does `ls science-it` work? Check the paths, there is probably something that doesn't exist. * Preciously no, but I creat science-it and now it works. Thanks! * is bash_profile a way to link your bash setting from local to e.g. triton? * `.bash_profile` is a script that is read and executed when a shell starts. Many of us have one which is shared between our local computers and Triton, to keep a consistent environment. But you have to find some way to copy it around yourself (e.g. scp) * since `bash_profile` and `.bashrc` are both bash scripts, you can have whatever logic and problams in them you want. Mine does different things on different systems. * How are umask and chown different from each other * umask: something that always exists, affects permissions of new files * chmod: changes permissions of existing files * chown: changes owners of existing files * Can we send files using the mail command * I see an option for it in the manual page, but I've never tested. Would be interesting to try. ------ To edit, click pencil icon at top. Always ask at very bottom (above this).