---
tags: ResBaz2021
---
# Introduction to Unix Shell: Automating the boring parts
## Monday, May 17th, 2021 3\:00-5\:00

[Back to Resbaz HackMD Directory](https://hackmd.io/@ResBaz21/directory)

## Workshop Summary

In this workshop participants will be introduced to;
* the Unix shell command-line,
* some essential command-line operations,
* navigating the file system using the shell,
* writing shell scripts to automate some of those command-line operations.

---
No prior command-line experience is required. Mac and Linux users will already have a Unix shell terminal installed. If you have a Windows machine, we ask you to download Git-Bash (https://git-scm.com/downloads, a Unix Shell for Windows) ahead of the workshop.

## Getting Started

### Introductions
- Patrick Bunn, UArizona PhD student in Hydrology and Atmospheric Science, [e-mail]
- Izzy Viney, UArizona MS student in Microbiology, [e-mail]
- Emilie Lefoulon, UArizona Postdoc in Microbiology, [e-mail].
- Jung Mee Park, University of Arizona, MLIS student library and information science at UA, PhD in sociology from Cornell, [e-mail], @jpark1917 (https://twitter.com/jpark1917)
- Rene Dario Herrera (they/them), University of Arizona Cancer Center, Data Analyst, [e-mail], [twitter@reneherrera](https://twitter.com/reneherrera), [github@renedarioherrera](https://github.com/renedarioherrera), [linkedin.com@renedarioherrera/](https://www.linkedin.com/in/renedarioherrera/)

### What is the Unix Shell?
* Most often we use a graphical user interface (GUI) to interact with our personal computer. Think of saving a document, you move the mouse to the save icon, or select save from the File tab. We use the mouse to navigate the GUI and then upon clicking, the GUI tells the computer to save.
* The Unix shell is different as we can use it to interact directly with the computer (it came before GUIs). The Unix shell is a command-line interface (CLI) allows you to control your computer using a set of specific commands. On the command-line you would type a command to tell the computer to save a document, essentially bypassing the mouse-navigation/click/GUI in the saving a document example above.
* The Unix shell is both a CLI and a scripting language. Creating scripts of these commands allows repetitive tasks to be done automatically and fast.
* It's one of the most underrated, yet useful, tools in computing (not many people include Unix shell commands in their acknowledgements unlike R, Python, or Matlab Packages)

### Why learn how to use the Unix shell?
* Quickly perform operations on large files (without reading into memory)
* Automation of repetitive tasks, with the correct commands this lessens chance of human error (need to rename 100 files? a Unix shell script makes light work of this task and do this later in this workshop)
* Enables the use of high-powered computers elsewhere, server or cloud-based (interacting remotely with using a Unix shell uses less bandwidth than when using a GUI)

# Opening a Unix Shell
**Windows**: Press the ⊞ Windows Key on your keyboard, then find Git Bash and open it.

**Mac**: Press ⌘ Command + Space Bar on your Mac Keyboard. Type in “Terminal” when you see Terminal in the Spotlight search list, click it to open the Terminal app.

**Linux**: Use Ctrl+Alt+T to open a Terminal or search for it in applications.

When the shell is first opened, you are presented with a prompt $, indicating that the shell is waiting for input:

  [username]$  

---
# Workshop Activities
We'll run through the most useful commands first, then depending on time create a shell script together to rename some files
## Essential commands

# Print username
whoami

# What time/date is it?
date

# Where in the computers files am I? 
# print working directory (the folder you a currently in)
pwd

# What kind of Unix shell am I using? (Bash is most common)
# echo is the command to print something on the commnad line.
# In this instance we print the default Unix variable SHELL and
# the $ and {} brackets are used to call a Unix variable
echo ${SHELL}

# We can create our own Unix variables using these commands, 
# but we need to use backtick `` to substitute in the date command
# as the variable learning_unix_since .
# Also, notice no spaces around = 
# this ensure it will be read as a unix variable
learning_unix_since=`date`
echo ${learning_unix_since}

# Use the up arrow ↑ on your keyboard to cycle back through the
# commands used so far (really useful to go back and try again 
# when something doesn't work)

___
## Some more essential commands
How is the pace so far...?

# Show the contents of a directory
ls

# Show the contents of a directory with details
ls -l

# See the contents of a subfolder 
# (the TAB key autofills, and is an essential tool!)
ls <subfolder>/

# Make a new folder with the mkdir command called: unix_workshop
mkdir <name>

# Change directory to specified pathway
cd <pathway>

# touch is a command that can be used to create files 
# if they don't already exist in the directory. 
# In the directory we just made unix_workshop, 
# create a text file using touch.
touch random_file_00.txt

# Change directory, up one in the pathway.
cd ../

# Wildcards 
* # one or more of any character
# Example use of * (list all files ending in .txt)
ls *.txt

# Make a copy of a file 
cp <existing filename> <destination path/filename>

# Move file, but also used for renaming
mv <filename> <destination>

# Remove a file; 
# use with caution
# use with caution
# use with caution
# use with caution
# use with caution
# use with caution
# use with caution
# there's no going back, it's gone forever!
rm <filename>

# Remove a directory;
# can only remove an empty directory
rmdir <name>

# Move to a directory with many different file types
# Get the line, word, character count of a file
wc <filename>

# Get just the line count
wc -l <filename>

# Pipe command; join commands together
# The pipe key is near Enter, 
# Shift+| (on US-keyboard) 
|

# E.g. count of items in the directory
ls <directory> | wc - l

# Get number of txt files in a directory
ls *.txt | wc -l

# Make a file with the output of a command using >
# will overwrite existing file
ls > <filename>

# Global regular expression
# searches for a string in a file
# good for quickly searching through 
# scripts to re-use bits of code
grep <"string"> <file>

# finding a file in your directory tree
find -name "<filename>"

# get help for any command 
<command> --help

# cancel incomplete command
CTRL + C

___
## View and change files

# Viewing the head (start) or tail (end) files
head <filename>
tail <filename>

# Let's be more specific about looking at the head of a file
head -n <no. rows> <filename>

# Open vi, a text editor (can use another text editor)
# this will create a file if it doesn't exist
vi <filename>

# To edit the file, go to EDIT mode by
# pressing Insert key, should see INSERT on the 
# bottom bar
Ins

# To go back to COMMAND mode, press Esc key
# bottom bar should go blank
Esc

# Exit vi, and save changes: in COMMAND mode
:wq

# Exit vi, without saving: in COMMAND mode
:q!

___
## Creating a shell script to change the names of some files
If we've covered enough for the day, we can skip to [Workshop Close Out](#Workshop-Close-Out) and those interested in creating a shell script can come back to here...

# create some files to eventually change the name of
touch random_file_{01..99}.txt

# the {01..99} creates a sequence from 1 to 99, try it after
# the  echo command.
# We should have just created 99 files empty text files, 
# using this sequence.

First we need to open a text file to write the script in.

# We can use vi to create the script
vi <filename>.sh 

This is the shell script that we'll go through together;

#!/bin/bash
# This first line ensures that your Unix shell knows 
# this is a bash shell script (not another shell type) 
# and can interpret it accordingly 

# First we set the pathway to the files wanting to be renamed
# we can use the present working directory
path_to_files=`pwd`

# Start a loop that looks for .txt files in the directory ${path_to_files}
# then loop over each file in that directory.
for file in ${path_to_files}/*.txt
do
  # Take the extension off the filename using % 
  # the % sign cuts a string of characters, like the filename
  # file in this instance
  prefix=${file%.txt}
  
  # use the prefix to create the new name for the file, 
  # by adding _modified to each filename
  outfile=${prefix}_modified.txt
  
  # print out the filename, the prefix and the new filename
  echo ${file} ${outfile}

  # perform the name change using the mv command, 
  # and the variable we just set up outfile
  # mv <filename> <new filename>
  mv ${file} ${outfile}
done

___
## Running a shell script

# to execute the shell script we just made
bash <filename>.sh

## Workshop Close Out

# see all commands since the start of the session
history

# save history to a text file
history > tutorial_history.txt

# clear your shell screen of previous commands
clear

> If you run commands with the shell, you can view your history of inputs: so you KNOW what you did. Whereas, a graphical interface does not leave behind a record of what happened.
---
## Questions and Answers
In this section, you can post your questions and feel free to answer if you have it. Questions will be answered during or after the workshop.

1. Ask your question.
- We'll answer it here
2. Your

---
### Links to additional resources
- This site is useful if you want to decode a command-line operation into its different components: https://explainshell.com/
:::info
**Session Feedback :mega:**

Use the link below to provide your feedback on the session:
[**Session Feedback Form**](https://forms.gle/TrnJpr9qRBEKdnVVA)
:::