# FIRST TRAINING SESSION

## Instructors:

- *Heather Andrews*, AE Faculty Data Steward
- *Bianca Giovanardi*, ASCM Assistant Professor

## Helpers:

- *Javier Gutierrez*, ASCM PhD candidate
- *Giorgio Tosti*, ASCM PhD candidate

## Description of the workshop

In this workshop we will cover the fundamentals of Bash, and exercise piping and scripting in Bash in order to make research data/code workflows automatic and efficient. Additionally we will create a Makefile showcasing what the Make utility does.

After this training session you will be able to navigate files and directories using Bash commands, and you will have the fundamental knowledge to start scripting in Bash and understand what Makefiles are used for.

## Program

| Time | Activity |
| ------------- | -------------------------------------------- |
| 09.30 - 09.45 | Introduction to the session |
| 09.45 - 10.35 | Bash fundamentals, piping and scripting |
| 10.35 - 10.50 | Exercise |
| 10.50 - 11.00 | Break |
| 11.00 - 11.15 | Searching |
| 11.15 - 12.05 | Makefile |
| 12.05 - 12.20 | Exercise |
| 12.20 - 12.50 | Macros |
| 12.50 - 13.00 | Feedback (mentimeter) |

#### Hello everyone :)

# Quick tips
- You can create a tree of directories at the same time with `mkdir -p dir1/dir2/dir3`. `-p` here stands for 'parents'.
- You can also use the `less` command if you want to visualize big documents in the terminal. Check it out! `less --help`
- Whenever you are uncertain about the command the flag `--help` is an always-working way to get documentation ;)
- When you're typing in a terminal, pressing `Tab` will do its best to autocomplete. A life and time-saver!
- Don't forget to `pwd` from time to time, to see where you're writing. Most terminals show it already, but you will really need it if you'll ever have to work in a 'bare' terminal.
- Do not leave spaces around `=` in bash scripts. Otherwise the `=` will be considered as a command (and result in an error)
- To see also hidden files in a directory (e.g. `.gitignore`), use `ls -la`
- In Python, if you have a work-in-progress function of which you know only the name, you can leave it there **without error**, by including the `pass` statement. E.g.

def plotting_file(list_files, out_file, x_text, y_text):
    pass

-

# Ask your questions here!
### Enclose your questions in between '______' and use a 'Q:' at the beginning of it, so that helpers know and can reply to your question more effectively. For example:
_______

Q: I get a 'command not found' error.
_______

Q: Seems like it works :-)
Indeed!
_______

Q: When I have the following in my bashrc file":
export PATH="/C/Users/luuki/Anaconda3/"

and reopen git Bash, the standard bash functions (like where) do not work anymore, but then python does work.

_______

R: Does anyone else have this issue?

I think the problem may be that you have not added the previous path at the end. Try out

export PATH="/C/Users/luuki/Anaconda3/:$PATH"

If it still does not work, we can fix it in the break. For now it's more important that your Git Bash is working with standard Bash commands

______

Q: Why can't we just use `wc -l data/input/AE*.csv` instead of `wc -l $(find data/input -name AE*.csv)`?

R: That does it too. The example is more to show how can you can use the output of one command as input variable for another one. As in to say "There are many ways to go to Rome".

______
Q: I do not have the .bash_history file... I do not know what I have wrong.

R: Have you checked in your home directory? Try `ls ~ | grep .bash_history`or `find ~/.bash_history`. Do you get an output?

Q: I get "no such file or directory".

R: can you do `cd` and then `pwd` and write the output below?

Q: From my home directory you mean?

R: yeah, if you just do `cd` without any argument, it should get to your home directory. Once you are there you can also do an `ls -la` and check manually if you see `.bash_history`.

Q: I only have these:
-rw-r--r-- 1 Gisela 197121 99 may. 19 17:43 .bash_profile
-rw-r--r-- 1 Gisela 197121 76 may. 21 10:45 .bashrc

R: hi Gisella, try creating a ~/.bash_logout file and there you can add the following line:

history > .bash_history

That will create a .bash_history file and it will dump the history in it.

------

# Extras:

solution to exercise:

mkdir code data/raw data/extra data/input data/output 

cd data 

for filename in *.csv 
do
    cp $filename raw/raw_$filename
done

cd ..

mv data/_R* data/raw

mv data/*.csv data/input

mv data/*.txt data/extra

plot_script.py

import sys
import matplotlib.pyplot as plt
import numpy as np


def plot_data (input_files, output_file, x_label, y_label) :
    plt.figure(figsize=(12,5))

    for file in input_files:
        data = np.loadtxt(file, delimiter=',')
        data_label = "Specimen " + file[-6:-4]
        plt.plot(data[:,0], data[:,1], label=data_label)

    plt.yscale('log')
    plt.legend()
    plt.xlabel(x_label)
    plt.ylabel(y_label)
    plt.savefig(output_file)
    plt.show()


if __name__ == '__main__':

    input_files = sys.argv[1:-3]
    output_file = sys.argv[-3]
    x_label = sys.argv[-2]
    y_label = sys.argv[-1]

    plot_data(input_files, output_file, x_label, y_label)

Makefile

.PHONY: plot_ae
plot_ae: data/output/Plot_AE_Specimen_1-8.png data/output/Plot_AE_Specimen_9-11.png data/output/Plot_AE_Specimen_12.png

# plot specimens tested under constant fatigue loading, no extra impact
data/output/Plot_AE_Specimen_1-8.png: data/input/AE*0[1-8].csv
	python code/plot_script.py data/input/AE*0[1-8].csv data/output/Plot_AE_Specimen_1-8.png "Time (s)" "Cumulative Energy (J)"

# plot specimens tested under constant fatigue loading, with in-situ impact
data/output/Plot_AE_Specimen_9-11.png: data/input/AE*09.csv data/input/AE*1[0-1].csv
	python code/plot_script.py data/input/AE*09.csv data/input/AE*1[0-1].csv data/output/Plot_AE_Specimen_9-11.png "Time (s)" "Cumulative Energy (J)"

# plot specimens tested under constant fatigue loading, with artificial drilling defect
data/output/Plot_AE_Specimen_12.png: data/input/AE*12.csv
	python code/plot_script.py data/input/AE*12.csv data/output/Plot_AE_Specimen_12.png "Time (s)" "Cumulative Energy (J)"

.PHONY: clean
clean:
	rm -f data/output/*.png

________________________________________________________________
________________________________________________________________

# SECOND TRAINING SESSION

## Instructors:

- *Heather Andrews*, AE Faculty Data Steward
- *Bianca Giovanardi*, ASCM Assistant Professor
- *Giorgio Tosti*, ASCM PhD candidate

## Helpers:

- *Javier Gutierrez*, ASCM PhD candidate

## Description of the workshop

In this workshop we will go through setting up a project structure, implement testing and continuous integration using the TU Delft Gitlab instance.

After this session you will have the fundamental practice to start integrating testing and continuous integration in your research project.

## Program

| Time | Activity |
| ------------- | --------------------------- |
| 09.30 - 09.40 | Introduction to the session |
| 09.40 - 10.00 | Recap and Makefile |
| 10.00 - 10.20 | Cookiecutter |
| 10.20 - 10.35 | Exercise |
| 10.35 - 11.15 | Project Structure |
| 11.15 - 11.25 | Break |
| 11.25 - 12.10 | Testing |
| 12.10 - 12.35 | Continuous Integration |
| 12.35 - 12.55 | Useful VSCode Tips |
| 12.55 - 13.00 | Feedback (Mentimeter) |

## IMPORTANT! Information for Continuous Integration exercise

**Name your Project**: "Project Fatigue 1" or "Project Fatigue 2" depending on which number you "are" (see list below).

**1s**: ES, LK, GT, AR
**2s**: GR, NS, XA, JG

# Ask your questions here!
### Enclose your questions in between '______' and use a 'Q:' at the beginning of it, so that helpers know and can reply to your question more effectively. For example:
_______

Q: I get a 'command not found' error.
_______

Welcome, everyone!

_______

# Extras
_____________

Cookiecutter Link

https://github.com/debrevitatevitae/cookiecutter-tud-ascm.git

_____________

Makefile

PLOT_SCRIPT=code/plot_script.py
RAW=data/input
PROCESSED=data/output
LANGUAGE=python
PLOT_EXE=$(LANGUAGE) $(PLOT_SCRIPT)

## plot_ae : Generates AE plots of all Specimens
.PHONY: plot_ae
plot_ae: $(PROCESSED)/Plot_AE_Specimen_1-8.png $(PROCESSED)/Plot_AE_Specimen_9-11.png $(PROCESSED)/Plot_AE_Specimen_12.png

# plot specimens tested under constant fatigue loading, no extra impact
$(PROCESSED)/Plot_AE_Specimen_1-8.png: $(RAW)/AE*0[1-8].csv
	@$(PLOT_EXE) $^ $@ "Time (s)" "Cumulative Energy (J)"

# plot specimens tested under constant fatigue loading, with in-situ impact
$(PROCESSED)/Plot_AE_Specimen_9-11.png: $(RAW)/AE*09.csv $(RAW)/AE_Specimen1[0-1].csv
	@$(PLOT_EXE) $^ $@ "Time (s)" "Cumulative Energy (J)"

# plot specimens tested under constant fatigue loading, with artificial drilling defect
$(PROCESSED)/Plot_AE_Specimen_12.png: $(RAW)/AE*12.csv
	@$(PLOT_EXE) $^ $@ "Time (s)" "Cumulative Energy (J)"

.PHONY: clean
clean:
	@rm -f $(PROCESSED)/*.png

.PHONY : help
help : Makefile
	@sed -n 's/^##//p' $<
    
    

_______________

TU Delft Research Software Policy

http://doi.org/10.5281/zenodo.4629662

_______________

Nice reference for more on how to write the `requirements.txt`:
https://medium.com/python-pandemonium/better-python-dependency-and-package-management-b5d8ea29dff1

_______________