<style>
.reveal .slides {
text-align: left;
}
</style>
# Essentials for every developers
<!-- Put the link to this slide here so people can follow -->
Slide: https://hackmd.io/@titaneric/ByXbnwa1P
Author: [titaneric](https://github.com/titaneric)
---
Introduce some tools and knowledges to level up the development and productability.
Inspired by [Missing semester in MIT](https://missing.csail.mit.edu/2020/)
---
## Setup the Linux working enviroments
- ssh
- mirror
- zsh
- anaconda
----
### ssh
```bash
ssh <user-name>@<host-ip> -p <ssh-port>
```
ssh config in `~/.ssh/config`
```
Host <host-name>
HostName <host-ip>
User <user-name>
Port <ssh-port>
```
```bash
ssh <host-name>
```
----
Authorized with public and private keys
```bash
# Work only in Unix
ssh-copy-id target-host
# Work in all OS including Windows 10
# generate the private/public key pair
ssh-key-gen
# need password
scp ~/.ssh/id_rsa.pub target-host:~
# need password
ssh target-host 'cat ~/id_rsa.pub >> ~/.ssh/authorized_keys'
# no more password
ssh target-host
```
----
### mirror
Fetch package more efficient
Change to near mirror site instead of default one
In Ubuntu, you could choose the following in NCTU
###### - NCTUCSCC - http://ubuntu.cs.nctu.edu.tw/ubuntu/
###### - NCHC - http://free.nchc.org.tw/ubuntu/
----
Instructions: [source](http://note.drx.tw/2012/01/mirror.html)
```bash
# find the current mirror site
$ cat /etc/apt/sources.list | grep main | awk '{ print $2 }' | \
cut -d'/' -f3 | sed -n '3P'
tw.archive.ubuntu.com
# replace to the selected mirror
$ sudo sed -i 's/tw.archive.ubuntu.com/free.nchc.org.tw/g' \
/etc/apt/sources.list
# update the cache
$ sudo apt update
```
###### - Echo the file content in sources.list, find the line matching `main`, print the 2'nd field (column), cut the line with delimeter `/` and look for 3'rd field (column), and print the 3'rd line at last.
###### - Replace the string matching `tw.archive.ubuntu.com` to `free.nchc.org.tw` globally.
----
### Extended bash - zsh
Depend on you to choose [zsh](http://zsh.sourceforge.net/) or [fish-shell](https://fishshell.com/).
Extremely recommend [oh my zsh](https://ohmyz.sh/) in zsh
installation: [source](https://github.com/ohmyzsh/ohmyzsh)
```bash
# install zsh
sudo apt install zsh
# install oh my zsh
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
# change default shell
chsh -s $(which zsh)
```
----
[theme](https://github.com/ohmyzsh/ohmyzsh/wiki/External-themes)
- powerlevel10k
[plugin](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins)
- [zsh-autosuggestions](https://github.com/zsh-users/zsh-autosuggestions)
- [zsh-completions](https://github.com/zsh-users/zsh-completions)
---
## Bash
[Shortcut](https://blog.ssdnodes.com/blog/cheatsheet-bash-shortcuts/) for Bash
###### - `Crtl + u`: clear the line
###### - `Ctrl + l`: clear the screen
###### - `Ctrl + a or e`: go to the begining/end of the line
###### - `Alt + f or b` or `Ctrl + < or >`: move one word (forward/backward)
###### - `Ctrl + w or d`: remove one word before/after the cursor
###### - `Ctrl + r`: search the history (could be combine with `fzf`)
----
### Useful command
- find, [fd](https://github.com/sharkdp/fd): find file/directory
- grep, [rg](https://github.com/BurntSushi/ripgrep): search lines matching pattern
- ls, [exa](https://github.com/ogham/exa): list directory
- cat, [bat](https://github.com/sharkdp/bat): echo the content of file
- head, tail: peek the file (useful dealing with large dataset)
----
- xargs: enable multiple command at a same time
- [fzf](https://github.com/junegunn/fzf): command-line fuzzy finder
- [fasd](https://github.com/clvv/fasd): quick access to files and directories
### Command for text processing
- sed
- awk
- cut
----
### command for training
- htop: monitor the CPU and RAM usage and process management
- watch nvidia-smi, [nvtop](https://github.com/Syllo/nvtop): monitor the GPU usage
- rsync: backups the files and directories remotely
- [drive](https://github.com/odeke-em/drive): backups the files and directories into Google Drive
----
### tmux
screen is also fine, but I love tmux
Scenario:
Imagine that you are a data scientist and your are training a large model on the remote host required amount of time, and you accidendally close the shell or your laptop is running out of the battery. What will happen and how to get rid of it?
----
Let's mimic the situation.
Start a new shell and write `long_duration.py`
```python
import os
pid = os.getpid()
i = 1
while True:
print(i, pid)
i += 1
```
Run it
```bash
python long_duration.py
```
----
When the shell is closed, the running process on the shell will be terminated.
tmux comes to help!!!
On bash, you could
```bash
# start a new tmux session
tmux
# list all sessions
tmux ls
# attach to specific session
tmux a -t 0
# remove specific session
tmux kill-session -t 0
```
----
In the tmux session, you could
###### - `Crtl + b + c`: start new window
###### - `Ctrl + b + ,`: Name current window
###### - `Ctrl + b + &` + `y`: Delete current window
###### - `Ctrl + b + p`: goto previeous window
###### - `Ctrl + b + n`: goto next window
###### - `Ctrl + b + <number>`: goto the number'th window
----
###### - `Ctrl + b + %`: Split the current window into two panes horizontally.
###### - `Ctrl + b + "`: Split the current window into two panes vertically.
###### - `Ctrl + b + <arrow>`: Focus on different pane
###### - `Ctrl + b + x` + `y`: Close current focused pane
###### - `Ctrl + b + d`: Detach current session
----
Now we try the feature of the tmux,
1. Start a new shell and new tmux session
```bash
tmux
```
2. Run the model
```bash
python long_duration.py
```
----
3. Detach the current session by `Ctrl + b + d` and search for pid for previous process in the htop.
4. Attach to previous session by
```bash
tmux a -t 0
```
Check the program is still running.
---
## Allow remote connection (in progress)
### [iptables](https://wiki.archlinux.org/index.php/Simple_stateful_firewall)
open port
```bash
sudo iptables -A INPUT -p tcp --dport <port-number> -j ACCEPT
```
### [ssh port forwarding](https://johnliu55.tw/ssh-tunnel.html)
#### local port forwarding
Scenario:
You do have right to login the ssh server but do not have right to open port on that server. Suppose you want to run tensorboard on that server to observe the model training by following commands
```bash
tensorboard --logdir logs/
```
Normally, tensorboard will run `locally` and listen to `6006` port. But how do you connect to tensorboard server outside the localhost server?
Local port forwarding comes to help!
This involve two machine, client and ssh server.
- Client is your computer responsible to run the local port forwarding and access tensorboard server via your favoriate browser.
- The ssh server is the server running the tensorboard locally (e.g., listen to `6006` port) and you have right to access that machine (e.g., listen to `22` port)
Command:
In your *client computer*, run the following command:
```bash
ssh -L <client-port>:localhost:<target-port> <ssh-host-name> -p <ssh-server-port>
```
In this scenario, you can run
```bash
ssh -L 9090:localhost:6006 <ssh-host-name> -p 22
```
It means run the command on the client side, open a local port forwarding (`ssh -L`), we want to connect to `<ssh-host-name>` with port `22`. Any connection to client side listening to port `9090` is equal to connecting to `<ssh-host-name>` with port `6006`. Note that `localhost` in the command is relative to `<ssh-host-name>` itself, because the tensorboard run locally on `<ssh-host-name>`.
If you run the command successfully, you will act like ssh to host normally. Then, open your favorite browser to `localhost:9090` to see you model training.
#### remote port forwarding
---
## Text editor - VS Code
It depends on you. Vim, Emux, Sublime are all fine.
Features of [VS code](https://missing.csail.mit.edu/2020/)
- cross-platform and open source
- debug, task, custom setting
- tons of extension
- remote development (extension)
----
### Remember the shortcut!!
Extremely helpful during devlopment!
----
### Recommended extension
#### Langauge
- Python
- Pylance
- C/C++
- Code Spell Checker
----
#### Highlight
- Bracket Pair Colorizer
- indent-rainbow
- Rainbow CSV
----
#### Misc
- Remote Development
- Settings Sync
- Excel Viewer
- Bookmarks
----
### Demo for remote development (in progress)
- Setup the ssh config
- Connect
---
### The beauty of Windows 10 features
VS code
built-in ssh
Windows Subsystem in Linux (WSL)
Windows Insider Program
Windows terminal
Windows Virtual Desktop
{"metaMigratedAt":"2023-06-15T10:44:52.384Z","metaMigratedFrom":"YAML","title":"Essentials for every developers","breaks":true,"description":"View the slide with \"Slide Mode\".","slideOptions":"{\"transition\":\"fade\"}","contributors":"[{\"id\":\"c852ef76-946d-4399-8d72-04000ee69fb4\",\"add\":12355,\"del\":3451}]"}