Automating Tasks with the Unix Shell

# Automating Tasks with the Unix Shell ## Important Information See the workshop website for more details: https://hifis-events.hzdr.de/event/19/overview Note that this pad is for taking collaborative notes and for voting/exercises. If you'd like to chat with the other learners in case you just missed out on something, please use the VC tool's chat for this. # Ice Breaker Exercises ... to get to know you ... and so you get to know the tools we use today! ## Exercise 1: Please enter your name and department below and share one thing about yourself. - Peter Steinbach, FWCC, I try to make neural networks dance as a day job - Lokamani, FWCC, I work on molecular electronics - Michael Nimtz, FWDH, I also have a beard ;) - Emil Fridman, FWOR, I'm doing nuclear reactor safety - Tobias Förster, HLD, I am pulsing magnets and calculate bandstructures - Alexander Schmeink, FWI, I like to ride my bicycle - Cecilia Contreras, FWGE, I should finish my PhD this year - Sibel Tas-Köhler, FWDF, I try to start to write my thesis nowadays - Theodore Papapetrou, FWDF, I work on particle mixing - Ashish Mishra, FWDH, I work on magnetorotational instability in DRESDYN Project. ## Exercise 2: Vote for something! Who is your favorite scientist from the list below? - [Lise Meitner](https://en.wikipedia.org/wiki/Lise_Meitner) +1 - [Marie Curie](https://en.wikipedia.org/wiki/Marie_Curie) `+1+1`+1 `+1` +1 +1 - [Wilhelm Röntgen](https://en.wikipedia.org/wiki/Wilhelm_Röntgen) - [Erwin Schrödinger](https://en.wikipedia.org/wiki/Erwin_Schrödinger) `+1` `+1` +1 For example, if you like Lise Meitner, put a `+1` behind her name to express your vote. ## Exercise 3: Break-out and discuss! We will shortly send you all into breakout rooms. Please discuss with the others why you chose the scientist from above and what relevance this person has to your career? Leave the breakout room after 3 minutes of discussion. # The Unix Shell ## getting ready Please drop a `+1` here once you are ready with the setup! TF: `+1` CC:`+1` AS: +1 FR: +1 EF `+1` MN`+1` TP `+1` ## Please share the output of `pwd` with us! PS: `/home/learner` AS: `/home/ryba/Schreibtisch/data-shell` CC: `/Users/contre63` TP: `/c/Users/Teo` ML: `/home/lokama42` MN:`/c/Users/nimtz27/Desktop/data-shell` TF:`/home/tobias` FR:`/home/florian` STK:`/c/Users/Sibel` AM: `/Users/AshishMishra` EF: `/d` ## More `ls` flags You can also use two options at the same time. What does the command ls do when used with the -l option? What about if you use both the -l and the -h option? AS: `ls -l` drwxr-xr-x 2 ryba ryba 4096 Aug 8 2019 creatures drwxr-xr-x 5 ryba ryba 4096 Aug 8 2019 data drwxr-xr-x 2 ryba ryba 4096 Aug 8 2019 molecules drwxr-xr-x 3 ryba ryba 4096 Aug 8 2019 north-pacific-gyre -rw-r--r-- 1 ryba ryba 86 Aug 8 2019 notes.txt -rw-r--r-- 1 ryba ryba 32 Aug 8 2019 pizza.cfg -rw-r--r-- 1 ryba ryba 21583 Aug 8 2019 solar.pdf drwxr-xr-x 5 ryba ryba 4096 Aug 8 2019 writing `ls -l -h` -> `ls -lh` ## Exercise: Absolute vs Relative Paths Starting from `/Users/amanda/data`, which of the following commands could Amanda use to navigate to her home directory, which is `/Users/amanda`? 1. cd . 2. cd / 3. cd /home/amanda 4. cd ../.. 5. cd ~ `+1` +´´ +1+1+1 +1 6. cd home 7. cd ~/data/.. `+1` 8. cd `+1` `+1` +1 +1 9. cd .. `+1` `+1` `+1` `+1` +1 +1 ## Relative Path Resolution Using the filesystem diagram below, if pwd displays `/Users/thing`, what will `ls -F ../backup` display? 1. ../backup: No such file or directory 2. 2012-12-01 2013-01-08 2013-01-27 3. 2012-12-01/ 2013-01-08/ 2013-01-27/ 4. original/ pnas_final/ pnas_sub/ +1 +1 +1 +1 5. +1 +1 +1 +1 `+1` ## Moving Files After running the following commands, Jamie realizes that she put the files `sucrose.dat` and `maltose.dat` into the wrong folder. The files should have been placed in the raw folder. ``` $ ls -F analyzed/ raw/ $ ls -F analyzed fructose.dat glucose.dat maltose.dat sucrose.dat $ cd analyzed ``` Fill in the blanks to move these files to the `raw/` folder (i.e. the one she forgot to put them in) ``` $ mv sucrose.dat maltose.dat ____/____ ``` STK: raw/sucrose.dat maltose.dat TP: `mv sucrose.dat maltose.dat ../raw` CC: mv sucrose.dat maltose.data raw/sucrose.data maltose.data FR: mv matose.dat sucrose.dat ./ raw/ TF: mv sucrose.dat maltose.dat ../raw AM: `mv sucrose.dat matose.dat ./raw` AS: mv sucrose.dat maltose.dat ./ raw/ MN: mv sucrose.dat maltose.dat ../raw/ solution: mv sucrose.dat maltose.dat ../raw ## Exercise: Renaming FilesST Suppose that you created a plain-text file in your current directory to contain a list of the statistical tests you will need to do to analyze your data, and named it: `statstics.txt` After creating and saving this file you realize you misspelled the filename! You want to correct the mistake, which of the following commands could you use to do so? 1. `cp statstics.txt statistics.txt` 2. `mv statstics.txt statistics.txt` 3. `mv statstics.txt .` 4. `cp statstics.txt .` CC: option 2 TP: 2 TF: 2 FR: 2 AS: 2 STK: 2 AM: 2 ## Exercise Wildcards When run in the molecules directory, which `ls` command(s) will produce this output? ```ethane.pdb methane.pdb``` 1. `ls *t*ane.pdb` 2. `ls *t?ne.*` 3. `ls *t??ne.pdb` 4. `ls ethane.*` AS: answer 3, not 1 (* can also be no characters) FR: 3 STK: 3 CC: 3 TP: 3 MN: 3 AM: 3 ## Exercise: Organizing Directories and Files Jamie is working on a project and she sees that her files aren’t very well organized: ``` $ ls -F analyzed/ fructose.dat raw/ sucrose.dat ``` The fructose.dat and sucrose.dat files contain output from her data analysis. What command(s) covered in this lesson does she need to run so that the commands below will produce the output shown? (a) ``` $ ls -F analyzed/ raw/ ``` (b) +1 ``` $ ls analyzed fructose.dat sucrose.dat ``` MN: mv *.dat /analyzed FR: mv *.dat analyzed TP: mv *.dat analyzed/ CC: mv *.dat analyzed/ AM: mv *.dat analyzed STK: mv *.dat analyzed/ AS: mv *.dat analyzed/ solution: `mv *.dat analyzed/` EF # Wrap-up Morning Session Please provide feedback on how you liked the morning session. For this, use the space below: ## Something that you want us to improve Note down anonymously what you were confused about or something that you want us to improve. - I was confused about the starting time. could have been earlier for me - - nothing really - fixed times for lunch published before registration opening of the course would be good, also, the times published under https://hifis-events.hzdr.de/event/19/page/1-video-conference-details and https://hifis-events.hzdr.de/event/19/timetable/#20200714 are different ## Something that you liked Note down anonymously what you learned or what you liked in the morning session. - The pace of the teaching. Having the chance to discuss within the small group. Doing the exercises also live with all the answers from the rest was very helpful to learn more. - the princples of teaching and the pace - we teach part of the carpentries' curriculum, carpentries.org - It's a good thing we also have access to the entire tutorial on github - HackMD is great - Although I am using the shell for some time I did learn new things. # Afternoon Session Will start at 1pm! ## sorting `lengths.txt` PS: fedora linux ``` $ sort lengths.txt 107 total 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 9 methane.pdb ``` CC: MacOS ``` $sort lengths.txt 9 methane.pdb 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 107 total ``` TF: Ubuntu 18.04.4 LTS 107 insgesamt 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 9 methane.pdb AM: MacOS $ sort lengths.txt 9 methane.pdb 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 107 total MN: Win10 $ sort lengths.txt 9 methane.pdb 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 107 total ``` AS (ubuntu linux):` 107 total 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 9 methane.pdb` FR: VirtualBox with Ubuntu 20: 107 insgesamt 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 9 methane.pdb TP: Win10 9 methane.pdb 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 107 total EF-Win 10 9 methane.pdb 12 ethane.pdb 15 propane.pdb 20 cubane.pdb 21 pentane.pdb 30 octane.pdb 107 total ## Exercise: Appending Data We have already met the `head` command, which prints lines from the start of a file. `tail` is similar, but prints lines from the end of a file instead. Consider the file `data-shell/data/animals.txt`. After these commands, select the answer that corresponds to the file `animals-subset.txt`: ``` $ head -n 3 animals.txt > animals-subset.txt $ tail -n 2 animals.txt >> animals-subset.txt ``` 1. The first three lines of animals.txt 2. The last two lines of animals.txt 3. The first three lines and the last two lines of animals.txt +1 +1 +1 +1 +1 +1 +1 +1 4. The second and third lines of animals.txt ## Exercise: Piping Commands Together In our current directory, we want to find the 3 files which have the least number of lines. Which command listed below would work? 1. wc -l * > sort -n > head -n 3 2. wc -l * | sort -n | head -n 1-3 3. wc -l * | head -n 3 | sort -n 4. wc -l * | sort -n | head -n 3 +1 +1 +1 +1 +1 +1 +1 +1 ## Exercise: Pipe Reading Comprehension A file called `animals.txt` (in the `data-shell/data folder`) contains the following data: ``` 2012-11-05,deer 2012-11-05,rabbit 2012-11-05,raccoon 2012-11-06,rabbit 2012-11-06,deer 2012-11-06,fox 2012-11-07,rabbit 2012-11-07,bear ``` What text passes through each of the pipes and the final redirect in the pipeline below? ``` $ cat animals.txt | head -n 5 | tail -n 3 | sort -r > final.txt ``` 2012-11-06,rabbit 2012-11-06,deer 2012-11-05,raccoon MN 2012-11-05,deer 2012-11-05,rabbit 2012-11-05,raccoon 2012-11-06,rabbit 2012-11-06,deer 2012-11-05,raccoon 2012-11-06,rabbit 2012-11-06,deer 2012-11-06,rabbit 2012-11-06,deer 2012-11-05,raccoon CC: 2012-11-06,rabbit 2012-11-06,deer 2012-11-05,raccoon TF: 2012-11-06,deer 2012-11-06,rabbit 2012-11-05,raccoon TP: 2012-11-05,raccoon 2012-11-06,deer 2012-11-06,rabbit AM: 2012-11-06,rabbit 2012-11-06,deer 2012-11-05,raccoonnan ## Exercise: Removing unneeded files Suppose you want to delete your processed data files, and only keep your raw files and processing script to save storage. The raw files end in `.dat` and the processed files end in `.txt`. Which of the following would remove all the processed data files, and only the processed data files? 1. rm ?.txt 2. rm *.txt 3. rm * .txt 4. rm *.* AS: 2 EF: 2 MN: 2 TP: 2 AM: 2 TF: 2 CC: 2 FR: 2 ## Exercise: Variables in Loops This exercise refers to the `data-shell/molecules` directory. `ls` gives the following output: ``` cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb ``` What is the output of the following code? ``` $ for datafile in *.pdb > do > ls *.pdb > done ``` Now, what is the output of the following code? ``` $ for datafile in *.pdb > do > `ls $datafile` > done ``` Why do these two loops give different outputs? CC: 1. cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb x 6 2. will show each of them file's name individually.. FR: 1 cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb 2. cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb EF: 1. cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb 2. cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb MN:1 6 times the list of all files 2 each of the files AM: 1. cubane.pdb methane.pdb pentane.pdb ethane.pdb octane.pdb propane.pdb cubane.pdb methane.pdb pentane.pdb ethane.pdb octane.pdb propane.pdb cubane.pdb methane.pdb pentane.pdb ethane.pdb octane.pdb propane.pdb cubane.pdb methane.pdb pentane.pdb ethane.pdb octane.pdb propane.pdb cubane.pdb methane.pdb pentane.pdb ethane.pdb octane.pdb propane.pdb cubane.pdb methane.pdb pentane.pdb ethane.pdb octane.pdb propane.pdb 2. cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb ## Exercise: Variables in Shell Scripts In the `molecules` directory, imagine you have a shell script called `script.sh` containing the following commands: ``` head -n $2 "$1" tail -n $3 "$1" ``` While you are in the `molecules` directory, you type the following command: ``` bash script.sh '*.pdb' 1 1 ``` Which of the following outputs would you expect to see? 1. All of the lines between the first and the last lines of each file ending in `.pdb` in the `molecules` directory 2. The first and the last line of each file ending in `.pdb` in the `molecules` directory +1 +1 +1+1 3. The first and the last line of each file in the `molecules` directory 4. An error because of the quotes around `*.pdb` +1 ## Exercise: Find the Longest File With a Given Extension Write a shell script called `longest.sh` that takes the name of a directory and a filename extension as its arguments, and prints out the name of the file with the most lines in that directory with that extension. For example: ``` $ bash longest.sh /tmp/data pdb ``` would print the name of the `.pdb` file in `/tmp/data` that has the most lines. AS: ``` # Usage: bash longest.sh directory extension wc -l "$1"*$2 | sort -n -r | head -n2 | tail -n1 ``` command: bash longest.sh ~/Schreibtisch/data-shell/molecules/ .pdb output: 30 /home/ryba/Schreibtisch/data-shell/molecules/octane.pdb EM ``` wc -l "$1"/"*$2" | sort -nr | head -n2 | tail -n1 ``` PS: (taken from AS) ``` # Usage: bash longest.sh directory extension wc -l "$1"/*$2 | sort -n -r | head -n2 | tail -n1 ``` # Wrap-up Afternoon Session Please provide feedback on how you liked the afternoon session. For this, use the space below: ## Something that you want us to improve Note down anonymously what you were confused about or something that you want us to improve. - ## Something that you liked Note down anonymously what you learned or what you liked in the afternoon session. -

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.