# Automating Tasks with the Unix Shell

## Important Information

See the workshop website for more details:
https://hifis-events.hzdr.de/event/19/overview

Note that this pad is for taking collaborative notes and for voting/exercises. If you'd like to chat with the other learners in case you just missed out on something, please use the VC tool's chat for this.

# Ice Breaker Exercises

... to get to know you ... and so you get to know the tools we use today!

## Exercise 1: Please enter your name and department below and share one thing about yourself.

- Peter Steinbach, FWCC, I try to make neural networks dance as a day job
- Lokamani, FWCC, I work on molecular electronics
- Michael Nimtz, FWDH, I also have a beard ;)
- Emil Fridman, FWOR, I'm doing nuclear reactor safety
- Tobias Förster, HLD, I am pulsing magnets and calculate bandstructures
- Alexander Schmeink, FWI, I like to ride my bicycle
- Cecilia Contreras, FWGE, I should finish my PhD this year
- Sibel Tas-Köhler, FWDF, I try to start to write my thesis nowadays
- Theodore Papapetrou, FWDF, I work on particle mixing
- Ashish Mishra, FWDH, I work on magnetorotational instability in DRESDYN Project.

## Exercise 2: Vote for something! Who is your favorite scientist from the list below?

- [Lise Meitner](https://en.wikipedia.org/wiki/Lise_Meitner) +1
- [Marie Curie](https://en.wikipedia.org/wiki/Marie_Curie) `+1+1`+1 `+1` +1 +1
- [Wilhelm Röntgen](https://en.wikipedia.org/wiki/Wilhelm_Röntgen)
- [Erwin Schrödinger](https://en.wikipedia.org/wiki/Erwin_Schrödinger) `+1` `+1` +1

For example, if you like Lise Meitner, put a `+1` behind her name to express your vote.

## Exercise 3: Break-out and discuss! We will shortly send you all into breakout rooms. Please discuss with the others why you chose the scientist from above and what relevance this person has to your career?

Leave the breakout room after 3 minutes of discussion.

# The Unix Shell

## getting ready

Please drop a `+1` here once you are ready with the setup!

TF: `+1`
CC:`+1`

AS: +1
FR: +1

EF `+1`
MN`+1`
TP `+1`

## Please share the output of `pwd` with us!

PS: `/home/learner`
AS: `/home/ryba/Schreibtisch/data-shell`
CC: `/Users/contre63`
TP: `/c/Users/Teo`
ML: `/home/lokama42`
MN:`/c/Users/nimtz27/Desktop/data-shell`
TF:`/home/tobias`
FR:`/home/florian`
STK:`/c/Users/Sibel`
AM: `/Users/AshishMishra`
EF: `/d`

## More `ls` flags

You can also use two options at the same time. What does the command ls do when used with the -l option? What about if you use both the -l and the -h option?

AS: `ls -l`
drwxr-xr-x 2 ryba ryba 4096 Aug 8 2019 creatures
drwxr-xr-x 5 ryba ryba 4096 Aug 8 2019 data
drwxr-xr-x 2 ryba ryba 4096 Aug 8 2019 molecules
drwxr-xr-x 3 ryba ryba 4096 Aug 8 2019 north-pacific-gyre
-rw-r--r-- 1 ryba ryba 86 Aug 8 2019 notes.txt
-rw-r--r-- 1 ryba ryba 32 Aug 8 2019 pizza.cfg
-rw-r--r-- 1 ryba ryba 21583 Aug 8 2019 solar.pdf
drwxr-xr-x 5 ryba ryba 4096 Aug 8 2019 writing

`ls -l -h` -> `ls -lh`

## Exercise: Absolute vs Relative Paths

Starting from `/Users/amanda/data`, which of the following commands could Amanda use to navigate to her home directory, which is `/Users/amanda`?

1. cd .

2. cd /

3. cd /home/amanda

4. cd ../..

5. cd ~ `+1` +´´ +1+1+1 +1

6. cd home

7. cd ~/data/.. `+1`

8. cd `+1` `+1` +1 +1

9. cd .. `+1` `+1` `+1` `+1` +1 +1

## Relative Path Resolution

Using the filesystem diagram below, if pwd displays `/Users/thing`, what will `ls -F ../backup` display?

1. ../backup: No such file or directory

2. 2012-12-01 2013-01-08 2013-01-27

3. 2012-12-01/ 2013-01-08/ 2013-01-27/

4. original/ pnas_final/ pnas_sub/ +1 +1 +1 +1
5. +1 +1 +1 +1
`+1`

## Moving Files

After running the following commands, Jamie realizes that she put the files `sucrose.dat` and `maltose.dat` into the wrong folder. The files should have been placed in the raw folder.

$ ls -F
 analyzed/ raw/
$ ls -F analyzed
fructose.dat glucose.dat maltose.dat sucrose.dat
$ cd analyzed

Fill in the blanks to move these files to the `raw/` folder (i.e. the one she forgot to put them in)

$ mv sucrose.dat maltose.dat ____/____

STK: raw/sucrose.dat maltose.dat
TP: `mv sucrose.dat maltose.dat ../raw`
CC: mv sucrose.dat maltose.data raw/sucrose.data maltose.data
FR: mv matose.dat sucrose.dat ./ raw/
TF: mv sucrose.dat maltose.dat ../raw
AM: `mv sucrose.dat matose.dat ./raw`
AS: mv sucrose.dat maltose.dat ./ raw/
MN: mv sucrose.dat maltose.dat ../raw/

solution: mv sucrose.dat maltose.dat ../raw

## Exercise: Renaming FilesST

Suppose that you created a plain-text file in your current directory to contain a list of the statistical tests you will need to do to analyze your data, and named it: `statstics.txt`

After creating and saving this file you realize you misspelled the filename! You want to correct the mistake, which of the following commands could you use to do so?

1. `cp statstics.txt statistics.txt`
2. `mv statstics.txt statistics.txt`
3. `mv statstics.txt .`
4. `cp statstics.txt .`

CC: option 2
TP: 2
TF: 2
FR: 2
AS: 2
STK: 2
AM: 2

## Exercise Wildcards

When run in the molecules directory, which `ls` command(s) will produce this output?

    1. `ls *t*ane.pdb`
    2. `ls *t?ne.*`
    3. `ls *t??ne.pdb`
    4. `ls ethane.*`

AS: answer 3, not 1 (* can also be no characters)
FR: 3
STK: 3
CC: 3 
TP: 3
MN: 3
AM: 3


## Exercise: Organizing Directories and Files

Jamie is working on a project and she sees that her files aren’t very well organized:

$ ls -F
analyzed/ fructose.dat raw/ sucrose.dat

The fructose.dat and sucrose.dat files contain output from her data analysis. What command(s) covered in this lesson does she need to run so that the commands below will produce the output shown?

(a)

$ ls -F
analyzed/ raw/

(b) +1 

$ ls analyzed
fructose.dat sucrose.dat

MN: mv *.dat /analyzed
    

FR: mv *.dat analyzed
TP: mv *.dat analyzed/
CC: mv *.dat analyzed/
AM: mv *.dat analyzed
STK: mv *.dat analyzed/
AS: mv *.dat analyzed/

solution: `mv *.dat analyzed/`
EF


# Wrap-up Morning Session

Please provide feedback on how you liked the morning session. For this, use the space below:

## Something that you want us to improve

Note down anonymously what you were confused about or something that you want us to improve.

- I was confused about the starting time. could have been earlier for me
-
- nothing really
- fixed times for lunch published before registration opening of the course would be good, also, the times published under https://hifis-events.hzdr.de/event/19/page/1-video-conference-details and https://hifis-events.hzdr.de/event/19/timetable/#20200714 are different

## Something that you liked

Note down anonymously what you learned or what you liked in the morning session.

- The pace of the teaching. Having the chance to discuss within the small group. Doing the exercises also live with all the answers from the rest was very helpful to learn more. 
- the princples of teaching and the pace
    - we teach part of the carpentries' curriculum, carpentries.org
- It's a good thing we also have access to the entire tutorial on github
- HackMD is great
- Although I am using the shell for some time I did learn new things.

# Afternoon Session

Will start at 1pm!

## sorting `lengths.txt`

PS: fedora linux

$ sort lengths.txt
107 total
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
9 methane.pdb

CC: MacOS

$sort lengths.txt
9 methane.pdb
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
107 total ```

TF: Ubuntu 18.04.4 LTS
107 insgesamt
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
9 methane.pdb

AM: MacOS

$ sort lengths.txt
9 methane.pdb
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
107 total

MN: Win10
$ sort lengths.txt
9 methane.pdb
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
107 total

AS (ubuntu linux):`
107 total
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
9 methane.pdb`

FR: VirtualBox with Ubuntu 20:
107 insgesamt
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
9 methane.pdb


TP: Win10
   9 methane.pdb
  12 ethane.pdb
  15 propane.pdb
  20 cubane.pdb
  21 pentane.pdb
  30 octane.pdb
 107 total


EF-Win 10
   9 methane.pdb
  12 ethane.pdb
  15 propane.pdb
  20 cubane.pdb
  21 pentane.pdb
  30 octane.pdb
 107 total




## Exercise: Appending Data

We have already met the `head` command, which prints lines from the start of a file. `tail` is similar, but prints lines from the end of a file instead.

Consider the file `data-shell/data/animals.txt`. After these commands, select the answer that corresponds to the file `animals-subset.txt`:

$ head -n 3 animals.txt > animals-subset.txt
$ tail -n 2 animals.txt >> animals-subset.txt

   1. The first three lines of animals.txt

   2. The last two lines of animals.txt

   3. The first three lines and the last two lines of animals.txt
        +1 +1 +1 +1 +1 +1 +1 +1
   4. The second and third lines of animals.txt
 
 
 
 ## Exercise: Piping Commands Together


In our current directory, we want to find the 3 files which have the least number of lines. Which command listed below would work?

  1.  wc -l * > sort -n > head -n 3

  2.  wc -l * | sort -n | head -n 1-3

  3.  wc -l * | head -n 3 | sort -n
 
  4.  wc -l * | sort -n | head -n 3
 +1 +1 +1 +1 +1 +1 +1 +1
 
 
 ## Exercise: Pipe Reading Comprehension

A file called `animals.txt` (in the `data-shell/data folder`) contains the following data:

2012-11-05,deer
2012-11-05,rabbit
2012-11-05,raccoon
2012-11-06,rabbit
2012-11-06,deer
2012-11-06,fox
2012-11-07,rabbit
2012-11-07,bear

What text passes through each of the pipes and the final redirect in the pipeline below?

$ cat animals.txt | head -n 5 | tail -n 3 | sort -r > final.txt

2012-11-06,rabbit
2012-11-06,deer
2012-11-05,raccoon

 MN
 2012-11-05,deer
2012-11-05,rabbit
2012-11-05,raccoon
2012-11-06,rabbit
2012-11-06,deer

2012-11-05,raccoon
2012-11-06,rabbit
2012-11-06,deer


2012-11-06,rabbit
2012-11-06,deer
2012-11-05,raccoon



CC: 
2012-11-06,rabbit
2012-11-06,deer
2012-11-05,raccoon

TF:

2012-11-06,deer
2012-11-06,rabbit
2012-11-05,raccoon

TP:
2012-11-05,raccoon
2012-11-06,deer
2012-11-06,rabbit

AM:
2012-11-06,rabbit
2012-11-06,deer
2012-11-05,raccoonnan 


## Exercise: Removing unneeded files

Suppose you want to delete your processed data files, and only keep your raw files and processing script to save storage. The raw files end in `.dat` and the processed files end in `.txt`. Which of the following would remove all the processed data files, and only the processed data files?

   1. rm ?.txt
   2. rm *.txt
   3. rm * .txt
   4. rm *.*

AS: 2
EF: 2
MN: 2
TP: 2
AM: 2
TF: 2
CC: 2
FR: 2


## Exercise: Variables in Loops

This exercise refers to the `data-shell/molecules` directory. `ls` gives the following output:

cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb

What is the output of the following code?

$ for datafile in *.pdb
> do
> ls *.pdb
> done

Now, what is the output of the following code?

$ for datafile in *.pdb
> do
> `ls $datafile`
> done

Why do these two loops give different outputs?






CC: 
1. cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb x 6
2. will show each of them file's name individually..


FR:
1  cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
 cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
  cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
   cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
    cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
     cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
     2. cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb

EF:
1.
cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
2.
cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb


MN:1 6 times the list of all files
   2 each of the files
   
    



AM: 
1. 
cubane.pdb	methane.pdb	pentane.pdb
ethane.pdb	octane.pdb	propane.pdb
cubane.pdb	methane.pdb	pentane.pdb
ethane.pdb	octane.pdb	propane.pdb
cubane.pdb	methane.pdb	pentane.pdb
ethane.pdb	octane.pdb	propane.pdb
cubane.pdb	methane.pdb	pentane.pdb
ethane.pdb	octane.pdb	propane.pdb
cubane.pdb	methane.pdb	pentane.pdb
ethane.pdb	octane.pdb	propane.pdb
cubane.pdb	methane.pdb	pentane.pdb
ethane.pdb	octane.pdb	propane.pdb

2.
cubane.pdb
ethane.pdb
methane.pdb
octane.pdb
pentane.pdb
propane.pdb



## Exercise: Variables in Shell Scripts


In the `molecules` directory, imagine you have a shell script called `script.sh` containing the following commands:

head -n $2 "$1"
tail -n $3 "$1"

While you are in the `molecules` directory, you type the following command:

bash script.sh '*.pdb' 1 1

Which of the following outputs would you expect to see?

  1.  All of the lines between the first and the last lines of each file ending in `.pdb` in the `molecules` directory
  
  2.  The first and the last line of each file ending in `.pdb` in the `molecules` directory
      +1 +1 +1+1 
  3.  The first and the last line of each file in the `molecules` directory
  
  4.  An error because of the quotes around `*.pdb` +1

## Exercise: Find the Longest File With a Given Extension

Write a shell script called `longest.sh` that takes the name of a directory and a filename extension as its arguments, and prints out the name of the file with the most lines in that directory with that extension. For example:

$ bash longest.sh /tmp/data pdb

would print the name of the `.pdb` file in `/tmp/data` that has the most lines.


AS: 

# Usage: bash longest.sh directory extension
wc -l "$1"*$2 | sort -n -r | head -n2 | tail -n1

command: bash longest.sh ~/Schreibtisch/data-shell/molecules/ .pdb

output: 30 /home/ryba/Schreibtisch/data-shell/molecules/octane.pdb


EM

wc -l "$1"/"*$2" | sort -nr | head -n2 | tail -n1

PS:
(taken from AS)

# Usage: bash longest.sh directory extension
wc -l "$1"/*$2 | sort -n -r | head -n2 | tail -n1

# Wrap-up Afternoon Session

Please provide feedback on how you liked the afternoon session. For this, use the space below:

## Something that you want us to improve

Note down anonymously what you were confused about or something that you want us to improve.

- 

## Something that you liked

Note down anonymously what you learned or what you liked in the afternoon session.

-