# Day 4 (Integrating Python and Shell for automation)

## Recap day 3

- `pathlib`, `shutil` - file management in Python
- `mkdocs` - to create website fast using markdown
- setting up apache and port forwarding
- automation for copying file directory
- automation for compiling `mkdocs` - `subprocess`
- automation transfer of files via `scp`
- (added git automation)

## Day 4

### Example for chaining process

find . -type f | grep .py$ | wc -l

This command chains `find`, `grep` and `wc` together. We use `subprocess.Popen()` to chains the commands. Example:

cmd2 = "grep py$"
grep = subprocess.Popen(
    cmd2, shell=True, 
    stdin=find.stdout,
    stdout=subprocess.PIPE
)

here `stdin` of the `grep` subprocess gotten the input from the `find` subprocess `stdout`, and the output of the `grep` subprocess is channeled to a new pipe.

Once the code is done, use the `scp2remote.py` that we created yesterday to copy the script to remote server for testing.

## Running ssh with sudo right

Example:

ssh -t -p 2233 dummy@127.0.0.1 "sudo apt install apache2"

# Threads and processes in Python

In Python we use `threading` and `multiprocessing` modules to run multi-threaded or multi-processes.

- if your task is I/O intensity -> use threads
- if your task is CPU intensity -> use processes

Process flow of using `threading` or `multiprocessing`

- prepare the task into a function `def fun(): ...`
- create threads/processes `t = threading.Thread`
- declare the target function `Thread(target=fun)`
- probably add your args tuple `Thread(..., args=(url,))`
- start the threads `t.start()`

Then the threads/processes will be detached from the main process and running concurrently. To request the threads/processes to join back the main program, call the `t.join()` method.

## Automation tips

- Automation only work if you know your workflow

Two common scenarios:

1. A workflow split into sub-tasks -> use `subprocess`
2. Same tasks repeated many times -> use threads