# Carpentries [Room 1]
---
## Links
### General
[Workshop website](https://nclrse-training.github.io/2021-04-26-NCL-ONLINE)
[Code of Conduct](https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html)
[Pre-workshop survey](https://carpentries.typeform.com/to/wi32rS?slug=2021-04-26-NCL-ONLINE)
[Post-workshop survey](https://carpentries.typeform.com/to/UgVdRQ?slug=2021-04-26-NCL-ONLINE)
[JupyterHub - databootcamp-1](https://databootcamp-1.nicd.app/)
### Day 1
[Downloadable data for day 1](https://swcarpentry.github.io/shell-novice/setup.html)
[Navigating Files and Directories information](https://davidherbert2.github.io/navigating_files_and_directories.html)
[Unix Shell Index information](https://davidherbert2.github.io/unix_shell_index.html)
### Day 1 Exercises
[Navigating Files and Directories](https://davidherbert2.github.io/navigating_files_and_directories.html)
[Working with Files and Directories](https://davidherbert2.github.io/working_with_files_and_directories.html)
[Pipes and Filters](https://davidherbert2.github.io/pipes_and_filters.html)
## Login
JupyterHub logins will be pasted here on the day of the workshop:
Use your student or staff number to log in. The first time you log in you need to enter a password. Pick any password. If you forget what your password is please let us know in the Zoom chat.
How To:
Go to your designated web-server in an internet browser. You will be prompted to enter a username - use your university one to log in. Once on there you can click the 'new' dropdown on the right hand side and select 'terminal' to start a shell session.
When in the shell, type (or copy/paste) :
wget https://swcarpentry.github.io/shell-novice/data/data-shell.zip
This will load in the files that we will be using during the shell carpentry session. This will initially be compressed in a zip, so to unzip type the following command:
unzip data-shell
NB: terminal/console/shell are all used for the same interface.
Using a terminal/ shell offline on your computer (i.e. not using jupyter-hub):
On linux or mac search for terminal in applications and you should be able to launch it straight from your applications list
On windows, you will need to use 'git-bash' or a similar port of the linux terminal (the windows has its own command line which uses different syntax). Instructions for installing this can be found: https://swcarpentry.github.io/shell-novice/setup.html
## Lessons
### Day 1 - Unix Shell (30 June 2021)
---
#### Notes
Shell Commands:
| Command | Description | Example |
| - | - | - |
| pwd | **Print Working Directory** | |
| ls | **List** files and directories in current working directory | |
| ls -F | Adds modifiers to more clearly identify different file types | |
| ls [directory] | List contents of [directory] | |
| help [command] | Show information about [command] | help ls |
| man [command] | Show **manual** page for [command] (more detailed than help, press q to exit) | man ls |
| cd [directory] | **Change Directory** to [directory] | |
| cd . | Change to current directory (does nothing) | |
| cd .. | Change to parent directory | |
| mkdir [directory] | **Make Directory** named [directory] | mkdir mydir |
| cp [file] [newfile] | **Copy** [file] to [newfile] | cp filename.txt newfile.txt|
| mv [file] [newfile] | **Move** [file] to [newfile] | mv filename.txt newfile.txt |
| rm [file] | **Remove** [file] | rm myfile.txt |
| rm -r [directory] | **Remove Recursively** [directory] and its contents (be **very** careful with this) | rm -r mydir |
| rm -rf [directory] | **Forcefully Remove Recursively** [directory] and its contents (be **very** careful with this) | |
| wc [file] | Show **Word Count** info of [file] (by default this is: line count, word count, character / count) | |
| wc -l | Show line count | |
| wc -c | Show character count | |
| wc -w | Show word count | |
| cat [file] | **Concatenate** the contents of [file] to the command line output (print contents of [file]) | |
| head -n [n] [file] | Show the first [n] lines of [file] | |
| tail -n [n] [file] | Show the last [n] lines of [file] | |
| sort -n [file] | **Sort** [file] by lines alphanumerically | |
| grep [pattern] [file] | Search for [pattern] in [file] | |
| find [directory] [file] | **Find** files named [file] in [directory] | |
| [command1] \| [command2] | **Pipe** (run) [command2] with output of [command1] | cat myfile.txt \| grep hello |
| [command] > [file] | Write output of [command] to [file] (overwrites an existing file) | |
| [command] >> [file] | Append output of [command] to [file] | |
| \*[file] | Any file name ending with [file] | \*.txt |
| [file]\* | Any file name beginning with [file] | myfile.\* |
| \*[file]\* | Any file name containing [file] | \*file\* |
| [command1] $([command2]) | Execute [command2] and use its' output as an input for [command1] | |
| for [variable] in [list]; do [command]; done | For every element of [list] (referred to using $[variable]) perform [command] | for file in \*.txt; do echo $file; done |
| for [variable] in [list]<br>do<br> [command]<br>done | Alternate for loop format using newlines instead of semicolons | for file in \*txt<br>do<br> echo $file<br>done |
Useful shortcuts:
- **Up/down arrow:** browse command history
- **Tab:** autocomplete current command or file name
- **Ctrl+C:** terminate current command
- (in nano) **Ctrl+O, Enter:** save file
- (in nano) **Ctrl+X:** exit nano editor
#### End of lesson survey:
What have we done well?
- I feel like everything was explained well
- Questions addressed really well
- The Exercises were went through thoroughly
-
What can we improve on?
- More polls and interaction
- More breaks/longer breaks
- More real life applications
- I miss some commands that output and fill the console screen because they were entered too fast
- Too fast I had to sit and learn for all of the breaks because I fall behind often
- The exercises notes were not easily accessible at all times (I think an email giving us links would be helpful)
### Day 2 - Version control with Git and programming with Python part 1 (1 July 2021)
---
#### Lesson 1 Notes
Git commands:
| Command | Description | Example |
| - | - | - |
| git config --global <span>user.name</span> [name] | Set git username to [name] | |
| git config --global user.email [email] | Set git email to [email] | |
| git config --global core.autocrlf input | Set correct default line endings (you may need a different command for Windows) | |
| git config --global core.editor "nano -w" | Set default editor to nano | |
| git config --global init.defaultBranch main | Set default branch name to main | |
| git config --list | Show list of changed settings | |
| git config --help | Show all available config options | |
| git init | Initialise working directory as git repository (delete .git directory to uninitialise git) | |
| git status | Show current repository status (mainly highlights differences from HEAD commit) | |
| git add [file] | Mark changes in [file] for commit | git add myfile.txt |
| git add [directory] | Mark all changed files in [directory] for commit | git add mydir |
| git add . | Mark all changes in repository (or current directory if not in repo root) | |
| git commit -m "[message]" | Commit all marked files to the repository and label this commit with [message] (the message should briefly explain what has changed) | git commit -m "New file added" |
| git log | Show list of past commits | |
| git log --oneline | Show commits with information condensed into one line | |
| git diff | Show all differences from the HEAD commit | |
| git diff [file] | Show all differences from the HEAD commit of only [file] | git diff myfile.txt |
| git diff --staged | Show all differences between marked changes and HEAD commit | |
| git diff [commit] [file] | Show differences in [file] since [commit] | git diff HEAD myfile.txt |
| git show | Combination of git diff and git log (can take the same arguments as git diff) | git show HEAD myfile.txt |
| git checkout [commit] | Reset repository contents to it's state from [commit] | git checkout HEAD |
| git checkout [commit] [file] | Reset contents of [file] to it's state from [commit] | git checkout HEAD myfile.txt |
| git checkout HEAD~[n] [file] | Reset contents of [file] to it's state from [n] commits before HEAD | git checkout HEAD~2 myfile.txt |
| git checkout [branch] | Switch to [branch] HEAD | git checkout mybranch |
| git checkout -b [branch] | Create new branch named [branch] and switch to it | git checkout -b mybranch |
| git branch [branch] | Create new branch called [branch] | git branch mybranch |
| git remote add origin [url] | Set remote to the repository at [url] | |
| git push | Push changes to remote | |
| git push origin [branch] | Push changes of [branch] to remote | git push origin main |
| git push --set-upstream origin main | Push changes to remote. After this you can use git push without arguments | |
| git pull | Pull changes from remote to local repository | |
| git clone [url] | Copy contents of repository at [url] into a local directory | |
| git clone [url] [directory] | Copy contents of repository at [url] into [directory] | |
Terminology:
- Commit Hash - A set of seemingly random characters used to uniquely identify each commit
- HEAD - Most recent commit (usually)
- gitignore - A file of the name .gitignore can be used to blacklist files and directories from being added to the repository
- github/gitlab/bitbucket - Websites for storing repositories remotely
- Remote - Reference to a remote repository on one of the above websites
#### Lesson 2 Notes
Python Types:
| Type | Description | Example |
| - | - | - |
| Boolean | True or false value | True<br>False |
| Integer | Whole number value | 50<br>2 |
| Float/Floating point | Decimal point value | 50.0<br>1.5 |
| String | List of characters (text) | "Hello World"<br>"50"<br>"Pi is 3.14" |
| List | Ordered sequence of values | [1, 2, "celery", 3.14, False] |
Python Operations:
| Operation | Description | Example |
| - | - | - |
| import [library] | Load contents of [library] for use in the program | import this |
| [var] = [x] | Set the variable named [var] to be equal to [x] | pi = 3.14<br>x,y = 5,6 |
| [x] + [y] | Add [x] and [y] | 5 + 10 |
| [x] - [y] | Subtract [x] and [y] | 10 - 5 |
| [x] * [y] | Multiply [x] and [y] | 2 * 4 |
| [x] / [y] | Divide [x] by [y] | 4 / 2 |
| [x] // [y] | Perform integer division (returns a whole number) | 4 // 2 |
| [x] ** [y] | Raise [x] to the power of [y] | 3 ** 2 |
| [x] < [y] | Return True if [x] is less than [y] otherwise return False | 2 < 4 |
| [x] <= [y] | Return True if [x] is less than or equal to [y] otherwise return False | 2 <= 4 |
| [x] == [y] | Return True if [x] is equal to [y] otherwise return False | 4 == 4 |
| [x] != [y] | Return True if [x] is not equal to [y] otherwise return False | 3 != 4 |
| [x] >= [y] | Return True if [x] is greater than or equal to [y] otherwise return False | 4 >= 2 |
| [x] > [y] | Return True if [x] is greater than [y] otherwise return False | 4 > 2 |
Python Functions:
| Function | Description | Example |
| - | - | - |
| print([message]) | Print [message] to the console | print("Hello World") |
| type([var]) | Returns the data type of [var] | type(10) |
| str([var]) | Convert [var] to a string (if possible) | str(10) |
| int([var]) | Convert [var] to an integer (if possible) | int("10") |
| float([var]) | Convert [var] to a float (if possible) | float(10) |
| help([function]) | Show detailed information about [function] | help(help) |
| [list].append([value]) | Add [value] to the end of [list] | |
| [list].pop([index]) | Remove and return value at [index] of [list] | |
| [list].insert([index], [value]) | Insert [value] into [list] at position [index] | |
Numpy Functions:
| Function | Description | Example |
| - | - | - |
| import numpy | Load numpy | import numpy as np |
| numpy.loadtxt(fname=[filename], delimeter=[delimeter]) | Load contents of the text file [filename] into a 2d numpy array and separate values by [delimeter] and newlines | numpy.loadtxt(fname="myfile.txt", delimeter=",") |
| [ndarray].dtype | Return data type of values in [ndarray] | |
| [ndarray].shape | Return shape (dimensions and size) of [ndarray] | |
| {ndarray}[{n1},{n2},...] | Return value in position {n1,2,...} of {ndarray} (number of ns corresponds to array dimensions) | myarray[0, 1] |
| {ndarray}[{n1}:{n2}] | Returns slice of {ndarray} from {n1} (inclusive) to {n2} (exclusive) | myarray[2:4] |
| numpy.max([ndarray]) | Returns the maximum value in [ndarray] | |
| numpy.min([ndarray]) | Returns the minimum value in [ndarray] | |
| numpy.std([ndarray]) | Returns the standard deviation of [ndarray] | |
| numpy.mean([ndarray]) | Return the average value of [ndarray] | |
Matplotlib Functions:
| Functions | Description | Example |
| - | - | - |
| import matplotlib | Load matplotlib | import matplotlib as mpl |
| import matplotlib.pyplot | Load pyplot | import matplotlib.pyplot as plt|
| matplotlib.pyplot.show() | Display loaded graphs | |
| matplotlib.pyplot.imshow([ndarray]) | Load a heat map of the 2D array [ndarray] | |
| matplotlib.pyplot.plot([ndarray]) | Load a line graph of the 2D array (axis 1 against axis 0) | |
| matplotlib.pyplot.figure(figsize=([width], [height])) | Create figure to put multiple graphs on | fig = matplotlib.pyplot.figure(figsize=(10, 4)) |
| [figure].add_subplot([nrows], [ncols], [index]) | Add subplot to [figure] and return an axis | ax1 = fig.add_subplot(1, 1, 1) |
| [axis].plot([ndarray]) | Add graph to [axis] | ax1.plot([[1,2,3],[10,20,30]])|
Time Functions:
| Function | Description | Example |
| - | - | - |
| import time | Load time | |
| time.ctime() | Return current time formatted as 'DAY MONTH DATE HH:MM:SS YEAR' | |
#### End of lesson survey (Lesson 1 - Version control with Git):
What have we done well?
- Good speed
- Nicely explained
- Helpers were really helpfull
- All questions were answered well
What can we improve on?
- Many technical issues wasted a lot of time
- Too long, during 8th hour my brain begun to shut down
-
#### End of lesson survey (Lesson 2 - Programming with Python part 1):
What have we done well?
- Most basic subjects were explained well and thoroughly
- Examples and exercises were very helpful
-
What can we improve on?
- A few basic subjects like if and else statements were forgotten?
- The topics learnt may be too basic
- Too long, during 8th hour my brain begun to shut down
### Day 3 - Programming with Python part 2 (2 July 2021)
---
#### Notes
Python Functions:
| Function | Description | Example |
| - | - | - |
| sorted([list]) | Returns alphanumerically sorted contents of [list] | |
| abs([num]) | Returns [num]. If [num] is negative it returns the positive equivalent | abs(-5) |
| for [element] in [list]:<br> [action] | Carry out [action] for every element in [list], where [element] is the value of the current element in [list] | for i in range(10):<br> print(i) |
| if [condition]:<br> [action] | Will perform [action] if [condition] is True | if 10 > 5:<br> print("10 is greater than 5") |
| if [condition]:<br> [action1]<br>else:<br> [action2] | Will perform [action1] if [condition] is True, otherwise will perform [action2] if False | if 2 > 5:<br> print("2 is greater than 5")<br>else:<br> print("2 is not greater than 5") |
| if [condition1]:<br> [action1]<br>elif [condition2]:<br> [action2]<br>else:<br> [action3] | If [condition1] is True perform [action1], otherwise if [condition2] is True perform [action2], otherwise perform [action3] | if 5 != 6:<br> print("5 is not equal to 6")<br>elif 5 != 5:<br> print("5 is not equal to 5")<br>else:<br> print("5 is equal to 5 and 6") |
| def [function]\():<br> [action] | Define a function named [function] which carries out [action] when called with [function]\() | def hello():<br> print("hello") |
| def [function]\():<br> [action]<br> return [value] | Define a function named [function] which returns [value] when called | def gimmepi():<br> return 3.14159
| def [function]\([parameter1], [parameter2], ...):<br> [action] | Define a function named [function] with parameters called with [function]\([input1], [input2], ...) | def add(a, b):<br> return a + b |
| def [function]\([parameter]=[default]): | Define a function named [function] which takes the optional parameter [parameter] which defaults to [default] when not specified | |
Boolean Operations:
| Operation | Description | Use |
| - | - | - |
| And | All conditions are True | True and True == True<br>True and False == False<br>False and True == False<br>False and False == False |
| Or | At least one condition is True | True or True == True<br>True or False == True<br>False or True == True<br>False or False == False |
| Not | Condition is inverted | not True == False<br>not False == True |
| Zero | Non zero numbers are true (including negative) | 0 == False<br>1 == True |
| Empty String | Non empty strings are true | "" == False<br>"Hi" == True |
| Empty List | Non empty lists are true | [] == False<br>[1, 2, 3] == True |
Glob Functions:
| Function | Description | Example |
| - | - | - |
| import glob | Loads glob | |
| glob.glob([filename]) | Returns a list of all files matching [filename] (wildcards can be used) | glob.glob("mydir/*.txt") |
#### End of lesson survey:
What have we done well?
-
-
-
What can we improve on?
-
-
-
Google Drive
Gist
Clipboard
Sign in with Wallet
