Try   HackMD

Linux Shell 2021 - Archive

tags: Linux Shell Basics

Welcome to Linux Shell Basics 2021

This is a hybrid course.

For the remote SSH to a native Linux

Day 1

Introduction

  • Write your question like this
    • an answer by someone who knows
    • A comment with a bit of code pasted
    • here longer code:
    ​​​​#!/bin/bash
​​​​whoami
​​​​hostname
​​​​str="hello"
​​​​echo $str
  • another question here
    • another answer
  • The link for 'Learning materials' does not work ?
  • Is Mac shell good?
    • Yes mac is linux and should use standard bash (not tested on very recent versions)
    • Could be zsh as well. Please type echo $SHELL and see what is the answer. If it is not bash, you can likely just start bash by typing bash.
  • What about MS powershell? is it good or should one stick with cgwin or similar?
    • Powershell is very different. Different language.
  • will you post the recording permanently somewhere?
    • Youtube if we are able to completely remove all faces and voices (except the lecturer :))
  • What is name of the book?
    • UNIX Power Tools, Shelley Powers etc, 3rd ed, O’Reilly

Hello world!

  • To test please click on "Edit" top right corner or click on the pencil

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    on top left (depending on your view.) To read without editing click on the eye icon
    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  • how do we sort out the stuffs / questions? or we randomly spam the questions here?

    • Randomly basically. We who support things in the backround can sort out things if needed. Just read and post always new stuff at the bottom.
    • If new things are added at the bottom, we know what changed and we answer. And everyone can answer of course, we are all learners :)

  • How can we open the github page. It does not work as is in the email: https://aaltoscicomp.github.io/linux-shell-scripting/

  • will you comment on bash variants like csh, bash, tcsh, bourne shell, etc ?

    • Briefly mentioned, but we will not go into the differences. .
    • tks. Sometimes Im a bit confuse if they are really different for most general usage or not.
    • There are two main types: "bourne shell" (sh, bash, many other) and c-shell (csh, tcsh, …). The differences between these are significantly different. Advanced shells like zsh have more features/some differences. Then some modern ones are more different.
    • But all basic use is similar. We teach bash because it's the most common base. If you use another, you can adapt these lessons easily enough.

  • what about MAc / *nix bashes. are they similar? did Apple change it substantially?

    • MAc uses zsh as its default shell since catalina
    • The shell itself is basically the same. Arrangement of files in the OS is different.

  • I use both Linux and Windows

    • You can use both even during the course if you like. Just get your shell working on either one.

  • I would like to use it for bioinformatic tools and I hope to get some answers to the basic questions

    • In this case we cover only basic terminal commands, any tool that works on the terminal will have a similar interface
    • And tools should come with some explanation as to input arguments etc. This course teaches you how to use those instructions

  • will the recording be send to us via email?

    • Yes you will receive a link via email.

Basic shell

https://aaltoscicomp.github.io/linux-shell/the-shell/

  • If you use Win10 and are on a current update, you can also use the windows command line to connect via ssh (in the same way as linux/mac) (Start -> CMD)

  • Does anyone else not see the full screen?

    • To me it looks full and correct
    • Maybe you zoomed on zoom?
    • In zoom there is "View Options" -tab (depends on your system, but usually that top of the screen). Double check you have there full-screen. And usually "side-by-side" mode (the default) works well.

  • Should i be understandng the SSH as something like a command version of our browsers (like chrome and firefox)

    • Sort of. It connects to remote computer and gives you interactive access there. No different from logging in to that computer locally - it's really great.
    • You do not really need to understand these (yet). But getting a pick picture of options available is the key point.

  • Do the exercises and demonstrations work under Ubuntu used from VirtualBox?

    • They should. +1

  • Another way to get a linux shell: Ubuntu applications in the Windows store

    • im using MobaXTerminal, and it looks as nice as the others.
    • This is essentially using Windows Subsystem For Linux.

  • can I add more packages in this gitbash like cygwin?

    • I think git bash is sort of standalone. We suggest it as a simple way to get some of the other options give you more packages.

  • Which one is better during this course for windowns users: putty and logging in to some linux server or git bash?

    • I think for practical purposes, any of them work well enough. Later you can find the one that works best for your work.
    • I'd suggest logging into a linux machine, if you have the option. git bash might lack some commands, that are normally available on a linux machine.

  • where to find the link for which terminal I'm using?

    • If you mean which version of a shell you use (on Linux/mac), you can use echo "$SHELL"
    • what does it means for what terminal Im using? Mine returned "/bin/bash"
      • That you are using bash (it points to the executable /bin/bash
    • There is "shell" (command line interface) and "terminal" (the windowed program that provides the text interface). Though often we use them interchangable. See above for the answer you probably want.

  • How can I get the SSH key?

    • You make it yourself, instructions depend on what ssh you are using (command line, PuTTY, etc)
    • Just to point out that this is more advanced question wrt to this course. Happy to answer still.
    • Which key do you mean? To log in to a linux machine via ssh? You would have to generate it yourself and either provide the admins of your linux server/cluster/machine with it (if password login is not allowed) or you need to login using your password and set it up yourself.

  • What do we have to do after we have donwloaded GitBash?

    • You need to install it if you plan using gitbash. If you have an affiliation with a university, most likely you can connect to you university Linux shell server "ssh MACHINE" even from the default Windows command line
      • ah, so we type this ssh username@turso.cs.helsinki.fi and we dont need to install this Gibash? This line is enough?
        • yes, assuming that turso.cs.helsinki.fi is a linux machine
          Image Not Showing Possible Reasons
          • The image file may be corrupted
          • The server hosting the image is unavailable
          • The image path is incorrect
          • The image format is not supported
          Learn More →
        • I used this before so should work.
        • Image Not Showing Possible Reasons
          • The image file may be corrupted
          • The server hosting the image is unavailable
          • The image path is incorrect
          • The image format is not supported
          Learn More →

  • The quote marks don't seem necessary with echo command

    • That depends on what you echo. If things get more complicated than simple text you might need them so using them is kind of best practice
    • thanks
    • try comparing echo Hello world to echo "Hello world" (note the extra spaces - hm, they don't show in the "view mode", but there are extra spaces between hello and world). discussion of the differences later!

  • how to get to the presemo page to answer

    • Just click on the answers

  • where? where is the link?

  • Why the new commands does not appear in: https://users.aalto.fi/~degtyai1/shell/bash_history.2021-11-08

    • you might need to reload the page

  • What's the sign to use, when you don't want to run the commands you typed in, but keep them in the history?

Starting out

https://aaltoscicomp.github.io/linux-shell/starting-commands/

  • I tried this commens du -hs * .[!.]* | sort -h but the ouptut is zsh: event not found: .] Why?

    • Perhaps that is a bash-ism and doesn't work in zsh. Checking… yes, seems like something zsh doesn't support (or more precisely, probably has a nicer looking syntax to do something similar)
      • Solution: put the .[!.]* in single quotes. Or the solution below.
    • There are indeed some differences how different shells interpret input like "[!.]". Here we are focusing on bash shell that is the most used. You can do the same command also on ysh but with minor edits. Essentially same functionality would in zsh works with command du -hs * .??*
    • please start bash first by typing bash
    • Want details? (advanced, don't worry about this): Shell parses the command line in different steps. ! means "replace the following part with something from the history". I guess bash does not interpert ! in the middle of a word, but zsh does.

  • if I put bash it says The default interactive shell is now zsh.

    ​​To update your account to use zsh, please run `chsh -s /bin/zsh`.
​​For more details, please visit https://support.apple.com/kb/HT208050.
​​bash-3.2$
    • if you type echo $SHELL what do you get?
      • /bin/zsh
    • This is essentially just a message by mac, that it uses zsh as default if you use any other pre-installed shell (like bash). I assume apple puts it in since they will use zsh for any support pages, and as we noticed, there could be commands that don't work on the other shell.

  • Regarding ZSH and BASH. If you do not have any preferences (or know what these are all about) just go with BASH. Most of the examples online are done with bash and you can work more easily with that.

    • So to install BASH on Mac?

  • I am using Gitbash and when I write man, it says that the command is not found. What should I do?

  • How to edit a part of the text here in HackMD as a code? But not the whole line.

    • backquotes `` around the part, like this: code (the code view shows this wrong)
  • In Gitbash is –help used instead of man?
    • --help is an option for many commands to print the help (it's a convention most programs use). man is something else, these are separate pages. Usually, --help or -h gives answers to short questions, and man more details. Since git-bash doesn't include all man pages, --help is a good alternative. --help is useful everywhere.
  • you can keep asking questions if you want
  • Perhaps you could mention to new users to NEVER run a bash command that you don't know what is doing. Specially when copy/pasting from online forums
    • one should have at least some general knowlegde of what the copied command is going to do in the machine.
  • If you have any general questions about how the course is running (questions, topics, breaks, etc), write them here and we will answer after the break.
  • Do the different enviroments affect the way the shell runs or what is the key difference between them?

Processes

https://aaltoscicomp.github.io/linux-shell/processes/

  • Portability There are quite a few commands that do not work (command not found) in Git BASH (e.g., man, top etc.)

    • Yeah. So, there are built-in shell commands. Then there are commands that interact with the oporating system. man, ps, etc are like this, and many more. This is because ps isn't a windows thing, and for man it doesn't include the pages.
    • so git-bash is reasonable to do some testing, but isn't quite a full working environment. If you do operate on Windows, it's fine but there are different commands to use.

  • My zoom process uses 120% CPU. Is that because it uses more than one core?

    • yep

  • How to end the ps auxw | less -S?

    • type q to quit the less program

  • How can we find out how strong a server is, when using putty, for example, howmany cores it has OR etc?

    • Many ways! These should work on linux (if these are installed): you could try lscpu and lsmem. cat /proc/cpuinfo, cat /proc/meminfo show other info.
      • How can I figure out what ressources are available to me on a server I connect to (e.g. if here is a restriction on how much a user can use)

  • I can't run htop or it is not a valid command. I use zsh shell on Manjaro Linux.

    • is this windows or git-bash? See an answer "portability" a little bit above.

  • The commands do not work in Git!

    • $ top
    • bash: top: command not found
    • see answer "Portability" above

  • Where do these external programs come from? Can we install them?

    • You can install them, but it's a bit of a hassle.
    • Really, they are part of the operating system. ps comes from Unix. man is more like an operating-system tool, not specific to bash. git-bash bundles just enough to run git. Tools like windows subsystem for linux, virtual machines, and Cygwin, run a whole unix operating system. We should improve the course some to consider this.

  • Maybe we can do a list of tools that don't work on Git bash and potentially suggest alternatives or link to instructions how to install them. +1

    • top ( suggestion on superuser )
    • A lot af the reason people use linux/mac isn't just the shell. It is the entire collection of these utilities that provide a powerful and scriptable environment.
      • I agree, and most likely, you will be on a linux server anyways when actually using the shell. Just thought of giving people some options. +1
      • From what I hear WSL (windows subsystem for linux) does all of these and makes windows first-class! (if only your university admins let you run it…)

  • If I detach a command from bash, does it break the parent/child relation? who is the new parent?

    • which way of detaching?
      • nohup XXXXX &
        • Parent child relation remains here.
          • Essentially, even if you close the respective terminal its process will be kept alive by the child
    • Normally the parent/child relation remains as is. Even if you leave the process in the background the parent still remains the same. There are some special cases where you can change this but those are really rare.
    • C-z (background) keeps parent-child. I think most things keep the relationship. Maybe everything, but the practical differences are things like "will killing the parent kill the child?" or "will sending a signal to the child signal the parent?". Then there are other operating system tools to control that properly.

  • Related to above, when detaching a ssh session, how do you keep leave stuff running?

    • Well, normally if you want to leave some process running on remote server then you use another handy program that allows this easily + allows you to return to this later (e.g. when going from work to home). Such programs are "tmux" and older alternative "screen". +1
    • If you detach something from ssh session this is similar to bash (ssh essentially runs bash on remote server for you). This needs the ssh session to remain active. Otherwise process is killed if ssh connection goes down.
    • The screen answer is good. Other tools handle detaching and keeping stuff running, find one and learn it well (otherwise you go deep into the OS). screen and tmux are standard.

  • screen is not available on GitBASH. The usefulness of screen is when you are logged in to a remote computer running Linux, so if you have the oportunity to connect to a remote Linux server, this is the ideal way to test the command Screen. Example ssh USERNAME@machine.xyz.fi e.g. for Aalto: ssh USERNAME@kosh.aalto.fi (other university users feel free to write down the name of your ssh servers here)

Exercises until xx:42

https://aaltoscicomp.github.io/linux-shell/processes/#exercises-1-1-1

  • Do as many as you can, some are dvanced. Continue with HackMD for questions

  • Sorry where is the exercise?, "processes" page

    • link above in the box

  • Does the system name mean the name and version of the operating system like Ubuntu 18?

    • Yes, operating system name. looks good.

  • I run manjaro linux and have zsh shell on default. I can run bash by typing 'bash' but still htop command is not working.

    • htop is an another program: it's newer, perhaps it needs to be installed. top is more likely to be there.
      • seems like it's not shipped with the os. you can use snap to install it though.
        • oh yeah, I can find htop with pacman from manjaro repos.

  • What's the difference between the root user and superuser (sudo)?

    • root is the user, sudo is program to change and run a single process as that user. There are other ways to do this: su (start bash as root), sudo (single command), log in as root directly.

  • how do I save the nano file?

    • Control-x then follow instructions (confirm file name and y). I do Control-x "y" "enter"
      • It did not work, I don't know how to confirm and where is it saved?
        • It should be saved in the folder you are in
    • hm. did you start nano with a filename like nano test.txt ?
      • ok, now it worked, but i don't know where the file was saved?
        • So you are out of nano again. can you type ls, and does it list the file?
          • it does
            • then you can e.g. type pwd which will show you your current working directory, and that's where the file was saved to
              • ok, thanks. if i want to save it somewhere else, how I would I do this?
                • if you already used a file name when starting nano, it will use that, otherwise if you just start nano, you can give a file name (including the full path) after you confirmed that you want to save it. if you don't giv an absolute path name (i.e. starting with /, it will assume its a relative path name based on the folder you are in), but all folders in the file name need to exist.

  • You can now change your presemo answers (hopefully)

  • Is it possible to copy a file from our Windows computer to the server when using "putty"?

    • This topic will be visited in detail a bit later. In short, not directly with putty. There are other multiple ways.
    • I know how to use other applications, e.g., Winscp, but I am not sure if it is also possibl via terminal.
      • Terminal is just a remote way to interact with (remote) computer. It does not connect your local machine to remote for things like file copy. Putty family of tools has another program (a bit like winscp) for that. Furthermore, depending where you operate you may not even need to do this at all. E.g. Univeristy HOME directory is the same for Windows/Linux systems. Enough to "use" that directly.
      • you can use e.g. the scp command (works the same as cp, but needs a server as part of the file name)

  • How do I quit a screen session?

    • Exit all programs within it (or there are probably Contral-a commands to quit and all processes within it it immediately)

  • what is the Stopped we get from jobs command?

    • It's in the background but suspended: it can't do any computation. If you then type bg it then it says "Running" which means it can do computation (and possibly write stuff to the screen).

  • What is the shortcut key to send a program to the background?

    • Control-z

  • I can't send nano to the background. It says suspension is not enabled?

    • what operating system? I think suspension in part depends on the OS and shell.
      • Manjaro
      • same with Git BASH
    • Sounds like suspending disabled by default on those systems, run 'nano -z …' to enable that ability
      • Alternatively, 'echo set suspend >> ~/.nanorc' and nano restart. That should set that option to default. See 'man 5 nanorc' for the details and other useful options.

  • I started mathematica from bash today and when I was done I tried to shut it down from the bash as well. When I started it from terminal the cursor moved to an empty line which implicated I was not able to type any commands. So I killed it with ^C and the the cursor moved to the next line and I was able to type commands again but the mathematica was still up and running? Also tried to send it to background and kill it there, the terminal said it was terminated but it still was running?
    Another question about this, why is it not possible to type commands after starting a program? The cursor is just blinking on an empty line. Is the idea to send it to the background if one wants to continue working in the same terminal?

Feedback day one

Your feedback is very important, we will adjust the next days (and next year!) based on it.

One good thing about today:

  • Basic command but still I like it
  • Good way to integrate both new users and more experienced ones.
  • Really nice to hear about why the shell and linux is working like it is and not only the commands.
  • Good to have enough time, so these things don't have be talked in a rush.
  • It was nice that there was a seperate channel where one could ask questions, so it didn't interrupt the teaching.

One thing to improve for next time:

  • Instead of nano lets switch to some other editing tool.
    • We must be generic here and use something that is doable by most of the participants, and I mentioned Emacs / Vim several times, but they are a subject for the separate tutorial
  • Going through the answers was way too fast to follow if trouble in making them
    • You mean the answers to the exercises?
    • yes
    • Ok, we will spend more time on the next ones
  • Perhaps cygwin would be a better suggestion for windows users because it is relatively easy to add more packages. Perhaps preparing a small tutorial on how to install and which packages users should also intall, e.g. moreutils, htop, etc
    • Good point, we will have it for the later courses in mind
    • Regarding Cygwin - to be tested yet - though the general point was, that priority #1 is an SSH connection to a native Linux installation, or VDI with ready to go Linux virtual machine. We can't recommend anything that would require a full guest OS installation, like virtualbox, then nothing that would require Administrator privileges, since we can't expect them. Cygwin and WSL are listed as an option, but that would be too advanced as a requirement for the Shell Basics course. But I'll add to the list anyway.
  • If possible, start the course at 1pm so that it is easier for us to synchronize with other university tasks and lunch time.
    • Not sure, we have had the SCIP courses since 2013, and by trying and failing have found that best time that suits most of the participants is either 9-12 or 12-15. But that was BPT (before pandemic times). We will discuss it internally once again, with Zoom, timing is somewhat more relaxed.
      • I agree with my fellow course participant above, the 12-15 slot is somewhat in the middle of the day and tends to reduce (at least my) productivity before and after. I would prefer a clear-cut morning (9-12, as you said) or afternoon (14-17, say) session, such that you can do something else in a productive way during the day as well. Plus, if you're taking the course remotely and have to cook your own lunch, starting the course at 12 is a bit early as well, since I then basically have to stop my morning work at 11 just to eat in time before the course.
        • Ok, we will see, let us discuss within the team
  • I think you could introduce bash with simples commands in the very start (listing directorires, wc, grep, time, date) before moving to 'administrative commands' like top, foreground/background, screen, kill, etc.)
  • The time management felt slighly off (lot of time in the beginning to open a shell, and then a rather quick walk through many process commands). Plus I think one break was missing? But maybe that's just a personal impression, no big deal :D +1, +1 for the time management
    • We planned one break in between, joined with the Exercise. Do we need two? Each hour?
  • I personally would have maybe liked a little bit longer break or two smaller ones to better keep up focus, especially with the time for doing the excercises being at the end of the lecture
    • Ok, two breaks sounds like a suggestion.

Day 2

Welcome back! Previous document is archived at https://hackmd.io/@AaltoSciComp/LinuxShell2021_archive

  • Questions here

    • and here an answer

  • What's the sign to use, when you don't want to run a command but want to keep it in the history?

    • #, essentially you are writing a comment, as in a bash script.

  • Will we also learn about the PATH and how to use it?

    • yes

  • Sorry were was the homework announced? I did not find it

    • I think he was more referring to homework for the teachers (i.e. updating information).
      • Potentially to general setup before the course, but nothing from Monday to now
      • ok Thanks for clarifying!

  • How we open a terminal? I have generated the keys using Putty now

Files and directories

https://aaltoscicomp.github.io/linux-shell/files-and-directories/

  • I need to link to server Narvi to use linux

    • See some of the links at the top of the page

  • I run ubuntu on an XFCE environment and the ctrl+t shortcut has never worked for me to open the terminal. Is there a reason for it?

    • Hm, I guess that is somehow a local configuration thing. If you go into the shortcuts you can probabyl figure out why.
    • Could it be a different hotkey, like alt or Win? Control-T is an important keystroke in terminals and other programs, so it would be surprising (to me) for it to be also used to open a terminal.
    • According to the https://defkey.com/xfce-shortcuts Ctrl + Alt + t, or just define on your own

  • Sorry, I dont see the command line in the zoom!

    • Maybe resize your Zoom window? Or adjust the display size (View Options -> Zoom ratio)
    • yes!
      Thanks
    • common problem! There are ways to configure zoom to make it better, but, well… better to solve it directly.

  • Hi, If there are 100s of folders and I have access to few of them. Can I search for a file (or location of that file) in all of the folders. For eg. I want to know if I have access to a certain file and where that file is.

    • Do you know the exact filename and path in advance?
    • Not the path, But I know filename
    • What I would do: find | grep the_filename
    • A more advanced person would avoid an unnecessary use of grep and do find -name the_filename. Find has options for almost kind of searching you may want, but I always hav((e to use the manual page to look it up!
    • Alright :) Thank you, it was very helpful.
    • was grep already covered? It prints lines matching a pattern. find [base_path] with no arguments prints all paths under that path. The magic of unix is combining simple programs to do complicated stuff.

  • with ln command, will the original file (file1) ie ln file1 file2 be deleted, will file2 also be deleted?

    • no (when there is no -s option). Hard links are like two independent pointers pointing to the same data.
    • Imagine hard links like an index in a book: two index entries pointing to the same page. You can remove either index entry, and the other one still works.
      • so they are like automatically synchronise file?
      • Not automatically sync. It literally is pointing to the same data (inode in technical terms)
    • To continue the analogy, with symbolic links, it's like an index entry says "see also" to the other index entry. If the pointed-to index entry goes away, the other one points to nothing.
    • Wikipedia:Hard_link has some diagrams and much more discussion - it is pretty cool stuff!
      • ohh ok, so the actual file will not be deleted unless both links are deleted!

  • when using ls -l command some files are green, some are blue, some are white and some have green background, then there is red background and I saw one with blinking red background, what does these mean?

    • Depending on your system ls information is enriched with colors. E.g. different types of files (directory, link, executable) may get different color coding. This is sometimes useful but not nesessary at all. You can check man ls for complete list of details.
    • Commonly: white - normal non executable file, blue - directory, green - executable file, not sure about red backgrounds (but I have seen it with lock files)
      • Blinking with symbolic links (ln -s) may mean that link exists but the target file is no longer present. So, may be an error/bad link. This varies a bit based on your operating system so.

  • I have number 17 as the number of the links in the list after command ls -l but I am pretty sure I don't have 17 links to that. Is the number increased when the file is copied?

    • What type of file is it? is it a directory?
      • yes
    • So each directory links to its parent directory (..) and itself (.). So directory link count is 2 (empty) + number of subdirs. Unix magic.

Break until xx:09

  • So chmod 000 would mean that there are no permission either for reading, writing and executing?

    • yes, noone could access those files. But root or their owner can still change the permissions.
      • So if reading = 4, all digits smaller than 4 won't give permission to read a file?
        • yes 3 is write and execute

  • What does umask 0002 mean?

Exercise 1.2, until xx:54

https://aaltoscicomp.github.io/linux-shell/files-and-directories/#exercise-1-2

  • Work on these as much as we have time to do, (*) is advanced.
  • HackMD for questions
  • Then we will go through the material

  • Would chmod 000 in my file prevent also me from reading/writing/Executing the file or just to other users in the same directory. (P.S in a shared directory)?

    • These permissions are per file (or directory) regardless of the parent file permissions. So 000 would essentially prevent reading/writing from everyone including yourself. Still, you have permission to re-edit these with chmod, so you can give your access back to the file assuming you are the owner of the file.

  • So what would be the command to keep the file accessible to only me to read/write/execute in a shared directory?

    • chmod 700 FileName, but others can change it back since they also "own" the folder. (Important comment below)
    • Well, if this is a shared directory, then most likely you should have read access to everyone there. Please always keep in mind what you are doing e.g. wrt data management.
      Image Not Showing Possible Reasons
      • The image file may be corrupted
      • The server hosting the image is unavailable
      • The image path is incorrect
      • The image format is not supported
      Learn More →
    • Can directory owners adjust permissions of other files within the directory?
      • It seems that the directory owner can delete the file, but not change its permissions or other properties.

  • How can I change to bin directory?
    It does not exist in my home directory

    • There are generl bin directories (like e.g. usr/bin or usr/local/bin) but you could also create a bin diretory in your home folder. Depending on your system settings you will have to add that path to your PATH variable manually (or in your .bashrc). How to do that will be discussed later. Without it being on your path you can only execute commands in the folder, while you are in the folder.
    • The home directory bin is generally made by each person if they want it, and not automatically there.

  • id -Gn command gives several groups but I actually don't know where these come from or what are these. How can I get some additional information about the groups?

    • Most unix systems have many default groups, and these contol things like ability to log in, audio/video devices, primary user, etc. A university system will add even more (controling access to other data and resources).
    • If it's a OS-default group, you could search "[OS name] [group name] group" and it might tell you what it's for.
    • example for Debian (probably also applies to Ubuntu): https://wiki.debian.org/SystemGroups

  • I have missed some parts of the instruction. When would we possibly get the access to recordings?

  • when we rm does the file go to bin?

    • If you use rm from the command line, there is no going back. Something that was deleted is deleted, no trashbin. The bin directory commonly refers to "binary" (executable) on unix systems.

  • I use Ubuntu. When I try

    ​​$ ls -l /usr/{sbin,bin}/[!a-zA-Z]

    the output is

    ​​ls: cannot access '/usr/sbin/[!a-zA-Z]*': No such file or directory
​​-rwxr-xr-x 1 root root 59736 Sep  5  2019 '/usr/bin/['
​​-rwxr-xr-x 1 root root    96 Mar  8  2021  /usr/bin/2to3-2.7
​​-rwxr-xr-x 1 root root 10104 Apr 23  2016  /usr/bin/411toppm
    • Short: that only means that /usr/sbin has no filename we are trying to list. That is ok.
    • Weird. I guess somehow that glob pattern [!a-zA-Z] isn't being parsed correctly, but it seems to somehow work too. Thinking about it…

  • Is ls -l ./* is correct command for task 3?

    • ls -ld .* should work as well, and avoids the odd "./." pattern in front of the files/folders
      • and, if we want to list real files/directories only, without '.' and '..', then ls -ld .[!.]*

  • How we can access one directory that you make for the course users in your home directory?

    • As an example
    • What do you mean?
    • The demo directory is on my local laptop, one can't access it without logging in. But, the use case is applicable to a server where many users are logged in.

  • I am still confused about the file permission in shared directory. If I am the one that created the file, is it so that now matter how I set the permission, the files will be accessible to anyone who has access to that directory? Or only by +s extension gives the permission?

    • If you create a file, then you are the owner. This means you can set the permissions for the file. If you set permission only to you then group+others cannot read/write to the file. But a big picture here is what makes sense (beyond bash). For shared directory most likely idea is that everyone has access. This is something to consider together with other users/research group.
      • Also, if you create a file in a directory that is owned by a group, that group can change the ownership of the file and subsequently change the access to a file. If we are talking about a "shared access" directory, you can only provide read access to others (common practice in a lot of universities) and not write access. In some universities, clusters had a general "all can read" policy (probabl not any longer due to GDPR), so that you can have a look at what others are doing.
    • Thank you I understood that. The idea is that I want others to see the output of my script, but I have my script in a file, that's the one I want to hide.
      • We found above that in shared directories, the directory owner does not have permission to change permissions of things inside the directory
        • Directory owner has permissions to change the ownership of the directory. Files underneath are either owned by the same user or someone else. This can be a bit tricky if there are indeed multiple users accessing and handling the same data. But in practice, for research case usually all those belong to the same group + ownership of every file should be read/write for the group (this is common practice at Aalto research case). So, yes you can do many "weird" things with the permissions. But commonly just do simple things that make sense.
          • this "research case" is not from Linux, but our own policies.
    • My recommendation: test it out and see. Linux is standerd but it is possible for other unixes to be slightly different
    • permissions break down to user/group/others read/write/execute. only user can adjust these. umask affects "default permissions"
    • Most of the rest that comes out is composed out of these concepts.

  • ls-ld .[!.]* doesn't work on my machine. The output is `ls: cannot access '.[!.]*': No such file or directory

    • that only means that there are dot-files in the current directory, thus ls has nothing to list. Try in your home, or 'touch .FileName' and try again.

    • Is this bash or zsh?

      • How to check this? I'm using Ubuntu.
      • echo $SHELL
        • /bin/bash

    • Is the directory empty? If it is, nothig matches the .[!.]* expansion. so it stays there literally, and you get that error message.

  • Ivan has a very small window for the terminal that only contains commands (w/o outputs). The commands do not displayed in full.

  • How can we install an exe file in one directory to be used?

    • Do you mean installing an executable file so that it can be called from anywhere?
      • yes
      • You would need to either copy the executable into a folder that is on the PATH or add the folder the executable is in to the PATH.
    • Basically an executable file can be anywhere, there is nothing special about where it is. To run from the current directory, use ./name. PATH is a seach path for other directories, which usually does not include the current directory ..

  • Can i use tar for .gz file? like something that write output for windows to read.

    • if it is a tar.gz yes, tar -xfz filename.tar.gz (-x is extract , -z is indicating that it is zipped, and -f indicates an archive file)
      • small note: instead of -z, one can have -a, and tar detects the compression algorithm on the fly, based on the extension
    • For graphical (GUI) interface there also tools that allow you to access compressed files. Not part of this course but if windows refer to graphical screen then most likely those are already there by default (Ubuntu, Mac, 7zip for Windows etc.).
      • I have outputs(.zip) that are in .gz in the UNIX platfor, and I wanted to extract a specific file in these .gz with the same name, so I am plannin got do it with command, rather than doing it in Window one by one.
      • so I suppose I will use zip in this case?
      • Just to clarify, gz and zip are two different formats of compressed files. .gz is quite common in linux side. For .gz you can use tar, for .zip there is command called ùnzip.
      • tar -xzf the_archive.tar dir/the_filename_to_extract.txt will extract out a single file from it. tar -tf the_archive.tar will list all the names.
      • so tar theoratically can deal with a wide form of zipping protocol?
        • yes and no. tar only bundles/unbundles the files, but it has options to call compression algorithms to pack/unpack the bundled file
        • ok thanks!
      • zip is compression + archive format. tar is archive format. gz is compression format. zip in (for practical purposes) is very separate from these two, I would not think to use one tool for the other.

Feedback, day 2

One good thing about today:

  • Now the course is becoming more interesting
  • I like the idea to start teaching from the basics
  • I finally learnt permissions and the overall knowledge of bash becomes structured in contrast to sporadic things I knew before.
  • .
  • .
  • .

One thing to be improved:

  • .
  • .
  • .
  • .
  • .
  • .

This is the end of the document, WRITE ABOVE THIS LINE ^^

HackMD can feel slow if more than 100 participants are editing at the same time: If you do not need to write, please switch to "view mode" by clicking the eye icon on top left

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
when will we get video of the course, todays

Last changed by 
 
AaltoSciComp
0
136

Read more

2025 summer kickstart - triton tutorial notes

This is the notes from the

Jun 3, 2025
Untitled

Here we watch TV together. https://www.twitch.tv/coderefinery

Jun 3, 2025
Conda

Modules come later. Now just load the module and mention what it does.

Jun 2, 2025
Introduction to Scientific Computing and HPC

(all times in EEST, Helsinki)

Jun 2, 2025
Read more from AaltoSciComp
Published on HackMD