---
tags: devtools2021
---
# Shell Scripting (Day 2 - Afternoon Session)
## Your current shell
You can find your current OS shell with the command `which $SHELL`. Few of the most widely used shells are bash and zsh. Since both zsh and bash are derived from the same **[Bourne](https://en.wikipedia.org/wiki/Bourne_shell)** shell family as bash does, most commands, syntax, and control structures will work just the **same**.
## Writing a shell script (30 mins)
Most users think that shell is equivalent to a *prompt* and a command line. That is in fact shell in **interactive mode**. A shell can also run in non-interactive mode, as when **executing** scripts. We can use scripts to **automate** certain logic, for example creating and deleting several files, renaming files, etc.
Scripts are basically **lists** of **commands** (just like the ones we can type on the command line), but **stored in a file**. When a script is executed, all these commands are (generally) executed **sequentially**, one after another
For example, here's a shell script that helps you to create a directory (if it doesn't exist already):
```bash=
#!/bin/bash
echo "Enter directory name"
read ndir
if [ -d "$ndir" ]
then
echo "Directory exist"
else
`mkdir $ndir`
echo "Directory created"
fi
```
Create a file called `createdir.sh` with the content above. Run it with the command `bash createdir.sh`.
You can also make the script directly executable because of the presence of the **shebang** line. Change its permission to be executable with `chmod` and then run it like an executable with `./<execname>`:
## Shebang
In computing, a shebang is the **character** **sequence** consisting of the characters number sign and exclamation mark (#!) at the beginning of a script.
When we try to run an executable file, the **[execve](https://man7.org/linux/man-pages/man2/execve.2.html)** program is called to **replace** the current process (bash shell if we are using terminal) with a **new** one and **decide** how exactly that should be done.
**If we try to run a text file** (scripts), execve **expects** the first two characters of the file to be “#!” (read “shebang” or “hashbang”) followed by a **path to the interpreter** that will be used to **interpret** the rest of the script. The most common need for using shebang appears when we’re writing shell scripts. Below is a simple example of a shell script that will simply greet a user:
```shell=
#!/bin/sh
echo "Hello, ${USER}"
```
> where `sh` is a command language interpreter that executes commands read from a command line string and `/bin/sh`is actually the path (**symbolic link**) to sh-compatible implementation of sh (Shell Command Language). In most cases, it’ll be **bash** (Bourne-Again SHell), but if we want to secure portability, we should use the symlink.
Also, remember that the script **must** be executable for this to work. When we create a new script file, we can make it **executable** using the `chmod` command:
```
chmod +x name_of_the_file
```
If we want to use different interpreters, such as `python`, we can change the shebang to any [ELF executable](https://linuxhint.com/understanding_elf_file_format/). For example, here we use `python` as the interpreter of the script:
```python=
#!/usr/bin/python
print("hello world")
x = 3
y = 4
print(x+y)
```
At first, the file pythonscript is not executable since we cant execute it (running `./pythonscript` results in error). After we changed the mode, we can now execute it using `./pythonscript`. This is functionally **equivalent** to running the command `python pythonscript`.
## Background knowledge (30 mins)
### File Permission
File permission can be viewed when you type `ls -l` command. The output can be something like this:
```
24120112 -rw-r--r--@ 1 natalie_agus staff 5492569 Sep 1 10:05 Develop and Collaborate (Day1 - Morning) - HackMD.pdf
23676798 drwxr-xr-x@ 9 natalie_agus staff 288 Sep 1 12:18 DevTools
```
We particularly want to bring your attention to 10-letter section of the output such as `-rw-r--r--` and `drwxr-xr-x`. What do you think they mean?
*Hint: Every file and directory on your Unix/Linux system is assigned 3 types of owner (User/creator, Group, Other). Each owner can have the right to execute, read, or write into the file.*
### File Links
In UNIX-like file systems:
* An inode is a data structure which purpose is to contain attributes about **all** files in the system.
* It also contains references to files (which otherwise is just an anonymous chunks of data) each given an **inode number**. You can try it by typing the command `ls -il` in your current directory to display the inode number of all your files in that path:
* A file in its entirety is **not** an inode, a file has an associated inode with it. inode contains only the file attributes (metadata).
It is pretty difficult for users to find their files based on inode numbers. Therefore, we have directories which allows us to traverse the file system using named paths.
* A directory is a **special** file whose content is some kind of **mapping** of names to files (more specifically inodes).
* A directory can have another directory as its entry (so we can search for files via recursion).
* We can have **different names** for the **same** file; in this case, they have no difference (in content) at all, like a person with different nicknames. These “extra names” mapped to the same inode entry are called **hard links**.
* There’s another type of link called **symbolic links**, also known as "shortcuts". A symbolic link is simply a file whose **content** is a **text string**(i.e: reference to another file or directory) that is **automatically** interpreted and followed by the operating system as a path to another file or directory.
### Example of File Link: Hardlink
* A file containing “helloworld” is created, with inode 24247396. We call it “input”.
* At this point, a directory entry named “input” pointing to this file containing “helloworld” is created within the directory named Links. We call this entry the **original** filename.
* We create a hardlink to “input”: `ln <inputfile> <outputfile>`
* Upon inspection, notice that **both** input and input_hardlink points to the same file: 24247396. There’s virtually **no difference** between the original name-file mapping “input” and “input_hardlink”.
Like regular files, path have path names in filesystem namespace, e.g: /Users/natalie_agus/Desktop/Links/input_hardlink
### Example of File Link: Symbolic link
You can create a symbolic link using the following command: `ln -s <inputfile> <outputfile> `
Note that a symbolic link has a new inode id (24247669 as shown). This means that symbolic link is created as a **new file** unlike hardlinks. If we delete the original reference `input.txt`, the symbolic link will be **broken**. However the hardlink will still work because the file is **not deleted on disk**.
## Bash script quick-tips
### Special characters
### Parameters
### Important control operators
The easiest way of performing a certain action depending on the **success** of a previous command is through the use of control operators. These operators are && and ||, which respectively represent a logical AND and a logical OR. These operators are used *between* two commands, and they are used to control whether the second command should be executed depending on the success of the first. **This concept is called conditional execution.**
**Yes, that space after the \[\[ and before the \]\] is intentional**. Maybe you wonder why it is \[\[ instead of \[? [Read this](http://mywiki.wooledge.org/BashGuide/TestsAndConditionals).
## Sourcing and Aliasing (30 mins)
The `source` command reads and executes commands from the file specified as its argument in the **current** shell environment. It is useful to **load** functions, variables, and configuration files into shell scripts (kind of like `import` in Python) or to the current shell environment.
You can source a file in two different ways:
```bash=
source FILENAME [ARGUMENTS]
. FILENAME [ARGUMENTS]
```
Bash **aliases** are essentially shortcuts that can save you from having to remember long commands and eliminate a great deal of typing when you are working on the command line. For example, you could set the alias `gpom` to be a shortcut for the `git pull origin master` command.
Creating aliases in bash is very straight forward. The syntax is as follows:
```
alias alias_name="command_to_run"
```
To test what it means as loading **into the current shell environment**, create a file called `config.sh``:
```bash=
VAR1="foo"
VAR2="bar"
alias gpom="git pull origin master"
```
Here we defined two variables, which otherwise doesn't exist in our original shell **session**. After we type the command `source config.sh`, the two variables are "registered" in our current session as shown:
However when we close our shell window and reopen a new one, we need to **reload** this `config.sh` script to maintain the loading of the two variables and the alias in the new **session**. It can be a tedious job having to repeatedly `source <FILE>` each time you want to load important stuffs like `$PATH` configuration, prompt customisations, aliases, etc.
## Bash Profiles (30 mins)
The Bash profile is a file on your computer that Bash runs every time a new Bash session is created. This is useful because we need to run certain code every time before starting to work. It has to be located in your home directory, with the name `.bash_profile` exactly.
Create a new file called `.bash_profile` in your EC2 home directory. You can use the GUI in Cloud9 or the following command:
```
touch /home/ubuntu/.bash_profile
```
Then paste the following content in `.bash_profile`:
```bash=
export VAR1="foo"
export VAR2="bar"
alias gpom="git pull origin master"
export PATH=/FundamentalDevTools:$PATH
```
Before having the `.bash_profile`, the shell doesn't know what `VAR1` and `VAR2`, or `gpom` are. The `PATH` also does not contain the new directory `/FundamentalDevTools`.
After having `.bash_profile`, we can successfully test for these new variables, alias, and `PATH` update each time the terminal starts:
## Running a Script on Startup with cron (30 mins)
The idea of a startup script is to help us get our system up and running again in the event of unexpected reboot, such as to re-run our node `server.js` upon startup.
Prepare a simple startup script called `startup.sh`. Take note of the path where you store it:
```bash=
#!/bin/sh
echo "Last reboot time: $(date). Running node server now." >> <absolute your path to log file>/logfile.txt
node <absolute your path to server file>/server.js
```
Remember to use **absolute path** rather than relative path. Then change it to be executable:
```
chmod +x startup.sh
```
The idea is to add a log each time the machine starts up into a file called `logfile.txt`, and then start the node server. The logfile is for us to know that the startup script is executed properly once the machine boots up.
### Cron
The software utility `cron` (also known as cron job) is a time-based job scheduler in Unix-like computer operating systems. We can tell it to do some stuffs like **running startup scripts** when the machine boots up. It is typically used by users who **set up** and **maintain** software environments. With cron, users can schedule jobs to run periodically at fixed times, dates, or intervals.
> If you want to learn more on how to write **cron schedule expression**, visit [this site](https://www.netiq.com/documentation/cloud-manager-2-5/ncm-reference/data/bexyssf.html) and [this site](https://crontab.guru) to get started. Go to [here](https://www.tecmint.com/11-cron-scheduling-task-examples-in-linux/) if you'd like to get started quickly without depth in understanding instead.
To add jobs to cron, type the command `sudo crontab -e`. This will bring up an editor where you can type the following command:
```
@reboot sh <path to your startup script>/startup.sh
```
Then you can reboot your system using the command:
```
sudo reboot
```
Check at start time that there's an entry in your `logfile.txt`:
```
Last reboot time: Thu Sep 2 12:22:52 UTC 2021. Running node server now.
```
Check that `node` is running `server.js` using the command:
```
ps aux | grep node
```
Using cron is quick and clean since we don’t have to deal with additional configurations, **but not every version of cron supports @reboot**.
To list out cron tasks for the current user, use the command:
```
crontab -l
```
To delete existing tasks, simply open the crontab editor again with `crontab -e` and delete the lines you no longer need.
## Exercise: Useful Tools (30 mins)
We now know how to write bash scripts, how to schedule jobs with cron, and how to read syntaxes of bash script. We also have learned a little bit about file links and file permission. The one thing left to do now is *practice* and familiarise ourselves with **useful commands** so we have a better idea on how to write a shell script that best suit our needs. Checkout the following commands by typing `man <command>` on your terminal and trying them out.
**Text-manipulation scripting tool:**
* awk
* sed
* grep
* sort
* uniq
* cat
* cut
* echo
* egrep
* fgrep
* wc
**Process monitoring commands:**
* ps
* top
* htop
* atop
* lsof
**Network diagnostics tools:**
* nmap
* tcpdump
* ping
* mtr
* traceroute
* dig
* iptables
* netstat
It takes more than a day to be able to write your own bash script, so take it easy! Start by reading great examples first, which can be found [here](https://github.com/awesome-lists/awesome-bash). They're useful tools that can make your development experience so much faster and useful. For instance, [this tool](https://github.com/sebglazebrook/aliases) creates useful aliases such as `gc` for `git commit`, `ll` for `ls -la`, and so on. To put the cherry on the cake, checkout `[Ohmyzsh](https://github.com/ohmyzsh/ohmyzsh)` to customise your `zsh` shell.
P.S.: checkout this awesome tool [Codegrepper](https://www.codegrepper.com). You can install it as a Chrome extension, then search for any dev-related queries, e.g: *"convert react to typescript"*, and **boom**! it shows the command for you:
Happy copy pasting! *Just ensure you know how they work*.
## Summary
This wraps up the first two days of the workshop. You have learned quite a bit of things in the short span of two days, namely:
* Understand how **Git** works in it's basic level as a source control tool, and tried out several essential Git commands that can be useful in any project
* Understand the **services** that are offered by an **Operating System** and some of its basic roles
* Introduced to the concept of Operating System **Kernel** and realised its importance in implementing the **network protocol stack**, **system security**, **communication** between processes, and **file system** management
* Understand the basic concepts of **Computer Network**: the 5-layer model, popular **transport** layer protocols, **IP** addressing, **NAT**, and **Domain**-name translation
* Recognize the importance of **Network Security**, made possible via **asymmetric** encryption and certificate authority (**CA**). Examples used include plain **HTTP, HTTPs (HTTP + TLS)**
* Basics of **bash scripting**, knowledge about **file links**, file **permission**, **cron** jobs, and useful **monitoring and development tools** in UNIX-based OS
As mentioned before, what we have covered barely scratched the surface about Operating System, Computer Networks, and Network Security due to our time constraint but hopefully they are sufficient to get you started and point you out to the direction that is most useful for your project or career. Please do not hesitate to contact us if you have any questions about these two days of workshop.
Take care and stay safe.
Sign in with Wallet
Connect another wallet