---
# System prepended metadata

title: The UNIX Programming Environment
tags: [self-learning]

---

---
title: The UNIX Programming Environment
tags: self-learning
---


[TOC]

> * basics: logging in and out, simple commands, correcting typing mistakes, mail, inter-terminal communication.
> * day-to-day use: file and the file system,directories, commonly-used commands.
> * the command interpreter or shell
## Chapter 1
- [time-sharing](https://zh.wikipedia.org/wiki/%E5%88%86%E6%99%82%E7%B3%BB%E7%B5%B1)
- [full duplex](): 允許通訊兩端可以在同一時間內互相傳輸訊號。
```
$ echo anything		//copies the character directly to the screen
```
- typing
   - RETURN key(**enter**):the end of a line of input
   - most of control characters do not have a key of its own like *enter*, they must be typed by holding down the CONTROL key(**ctrl**)
     - BREAK/DELETE/ctrl-c: stops a program immediately
     - Backspace/**\#**(sharp):erases the last character typed,each **#** erases one more
     - **\@**(at)/ctrl-u:line kill
     > To erase a backsplash: \\##
### Logging in
- get a login name and password from your system administrator
- prompt: **$** or **%**, indicating that the system is ready to accept commands from you
### date
```
$ date
Tue Jul  7 22:56:55 CST 2020
```
### who
- tells you everyone who is currently logged in
```
$ who
s107321014 pts/0        2020-07-07 21:43 (59.126.222.231)
// user name/system's name for the connection being used
$ who am i
```
### Strange terminal behavior
- fix this by turning the terminal off and on, or by logging out and logging back in
- Read the description of the command **stty**(set terminal options)
> tty -> teletype , an archaic synonym for "terminal"
```
$ stty -tab 	//convert tabs into right number of spaces
$ tabs terminal-type
```
### Stopping a program
- **BREAK/DELETE/ctrl-c**: stops whatever the program is doing but leaves you in that program
- **ctrl-s**: just want output to pause, for example to keep something critical from disappearing off the screen, the program will suspend until you type **ctrl-q** to resume
### Logging out
- **ctrl-d**: tells the shell there is no more input
### mail
* read your mail:
```
$ mail
```
* write a mail:
```
$ mail s107321014@mail1.ncnu.edu.tw
type something, and type a ctrl-d after the last line
[ctrl-d]
$
```
* if you change your mind half-way through composing the letter, press DELETE instead of *ctrl-d*, it will be stored in a file called **dead.letter**
#### write
* two-way communication path:
```
$ write mary
type something and ending each turn with (o)(means over)
or type (oo)(means over and out), then type ctrl-d
```
### news
```
$ news
```
- *netnews* and USENET
### man(the manual)
- describes most of what you need to know about the system
```
$ man command
```
### learn/teach
- provides computer-aided instruction on the file system and commands
```
$ learn
```
### ed(editor)
- create a file called **junk** 
```
$ ed		//
a		//append,add text
now type in
whatever text you want...
.		//stop adding text
w junk		//write your text into a file called junk
39		//prints number of characters written
q		//quit editor
```
- access it again by typing:
```
$ ed junk
```
### ls
- list, add an initial minus sign(-) and a single letter to alter its default behavior
```
$ ls -t		//listed in time order, most recent first
$ ls -l		//long listing, more information
$ ls -u		//gives information on when files were used
$ ls -lut		//long listing in the order of most recent use
$ ls -rt		//listed in the reverse order of time
```
### permission
- **-rwxr-xr-x 1 owner**
  - first: type of file
    - **d**: directory
    - **l**: link
  - -___ ___ ___(-owner/group/others)
  - **r**: read
  - **w**: write
  - **x**: execute
  - **1**: number of links to the file
### cat/pr
- ed
  - only print one file at a time
```
$ ed junk
19		//ed reports 19 characters in junk
1,$p		//print lines 1 through last
text in junk
blablabla
q		//all done
$
```
- cat(catenated)
  - you should quickly type *ctrl-s* to stop output from *cat* before it flows off your screen
```
$ cat junk temp		//print both junk and temp
text in junk
blablabla
text in temp
blablabla
$
```
- pr
  - in a form suitable for line printers: every page is 66 lines long, with the page number, filename, date and time that the file was changed
  - after a file was printed, it will skip to the top of a new page and print the next file
```
$ pr junk temp
(line 1, blank)
(line 2, blank)
2020-06-02 00:33           junk           Page 1
(line 4, blank)
(line 5, blank)
text in junk
blablabla
(more blank lines until it has 66 lines)
(line 1, blank)
(line 2, blank)
2020-06-02 00:33           temp           Page 1
(line 4, blank)
(line 5, blank)
text in junk
blablabla
(more blank lines until it has 66 lines)
$
```
-   - produce multi-column output, but not a formatting program
```
$ pr -3 filenames		//prints each file in 3-column format
$ pr -m filenames		//prints a set of files in perallel columns, side by side
```
> see **nroff** and **troff**(formatters) in chapter 9
### mv
- rename the file, moving it from one name to another
- if the new name is already exists, the target file is replaced
```
$ mv junk precious
```
### cp
- make a copy(have two versions of something)
```
$ cp junk junk.save
```
### rm
- remove(make the file nonexistent)
```
$ rm junk junk.save
```
### limits of filename
* <=14 characters
* had better use only letters, numbers, period, and underscore
### wc
- word-counting: counts the lines, words, and characters(**word** means any string of characters that doesn't contain a blank, tab or newline)
```
$ wc poem
	8	46	263 poem
```
### grep
- g/regular-expression/p
- searches files for lines that **match or don't match** the pattern
```
$ grep fleas poem	//look for the "fleas" word in the file called poem
line which has "fleas"
line which has "fleas"
line which has "fleas"
blablabla
$
```
```
$ grep -v fleas poem	//inverting the sense of the match
line which has no "fleas"
line which has no "fleas"
line which has no "fleas"
blablabla
$ 
```
* see more about *grep* in [chapter 4](https://hackmd.io/I_szuKx9Q_-zhrSCeSVzcA?both#Chapter-4)
### sort
- sorts its input into alphabetical order line by line
```
$ sort poem
blanks first
A->Z(upper case letters)
a->z(lower case letters)
```
```
$ sort -r	//reverse normal order
$ sort -n	// sort in numeric order
$ sort -nr	// sort in reverse numeric order
$ sort -f	//fold upper and lower case together
$ sort +n	//sort starting at (n+1)-st field
```
* see more about *sort* in [chapter 4](https://hackmd.io/I_szuKx9Q_-zhrSCeSVzcA?both#Chapter-4)
### tail
- prints the last 10 lines of a file
- the number of lines can be specified
```
$ tail -1 poem	//prints the last line of poem
$ tail +3 poem	//prints with the 3rd line
```
### cmp
- comparing files
- finds the first place where two files differ
- doesn't say what the difference is, nor does it identify any differences beyond the first
```
$ cmp poem new_poem
poem new_poem differ: char 58, line 2
$
```
* use when want to be sure that two files really have the same contents
* works on any types of files
### diff
- comparing files
- reports on all lines that are changed, added, or deleted
```
$ diff poem new_poem
2c2	//the line 2 in poem has to be changed into line 2 of new_poem
<	upon their backs to bite 'em,
---
>	upon their backs to bite them,
4c4
<	and so ad infinitum.
---
>	and so on ad infinitum.
$
```
* use when the files are expected to be somewhat different, and you want to know exactly which lines differ
* works only on files of text
### shell
- user <--> **shell** <--> kernel
- an ordinary program
  - filename shorthands
  - Input-output redirection
  - Personalizing the environment
#### Filename shorthand
1. ==*== means any string of characters
```
$ pr ch*	//matches all filenames in the current directory that begin with "ch"
```
   - It's not a property of the command, but a service of the shell
```
$ wc ch1.*
    113   562	 3200  ch1.0
    935  4081   22345  ch1.1
	(blablabla)
   3801 16210   88933  total
$
```
   * use with *echo* can lists the names of all the files in chapter1
```
$ echo ch1.*
$ echo *	//lists all the filenames in the current directory in alphabetical order
```
  - **\*** can be use in any where and occur several times
```
$ rm *.save	//removes all file that end with .save
```
2. ==[...]== matches any of the characters inside the ==brackets==
  - a range of consecutive letters or digitss can be abbreviated
```
$ pr ch[12346789]*	//print chapter 1,2,3,4,6,7,8,9 nut not 5
$ pr ch[1-46-9]*	//same thing
$ rm temp[a-z]	//remove any of tempa,...,tempz that exist
```
3. the ==?== pattern matches any single character
```
$ ls ?	//list files with single-character names
$ ls -l ch?.1	//list ch1.1 ch2.1 ch3.1, etc. but not ch10.1
$ rm temp?	//remove files temp1,...,tempz, etc.
```
:::info
you cannot make up new filenames by using patterns, those matches no existing filenames
:::
* Those patterns can be used in pathnames
```
$ wc /usr/mary/*
$ wc /usr/*/calendar
```
* enclose the entire argument in single quote if you want to precede a special character, or with a backslash
```
$ ls '?'
$ ls \?
```
#### Input-output redirection
1. ==>== means "put the output in the following file, rather than on the terminal"
 - the file will be created if it doesn't already exist, or the previous contents overwritten if it does
```
$ ls >filelist	//the list of filenames will be placed in the file called "filelist"
```
 - combine several files into one by capturing the output of *cat* in a file
```
$ cat f1 f2 f3 >temp
```
2. ==>>== operates much as ==>== does, it means "add to the end of"
```
$ cat f1 f2 f3 >>temp	//copy onto the end of whatever is already in temp, instead ofoverwriting the existing contents
```
3. ==<== means to take the input for a program from the following file
```
$ mail mary joe tom bob <let	//prepare a letter in file let, then send it to several poeple
```
* blanks are optional on either side of > or <
4. combine commands to achieve effects not possible otherwise
```
$ who >temp	//prints one line of output per logged-on user
$ sort <temp	//prints an alphabetical list of users
$ wc -l <temp	//counts lines, so you can counts users
$ grep mary <temp	//see if mary is logged on
```
```
$ ls >temp
$ wc -1 <temp	//counts the files in current directory
$ pr -3 <temp	//print the filenames in three columns
```
* centralizing the facility in the shell: the interpretation of > and < is being done by the shell
```
$ sort <temp	//interpreted by the shell, sort doesn't see the filename temp as an argument, it instead sorts its standard input
$ sort temp	//same thing but a little different
```
* **given a list of filenames** v.s. **no filenames are given**
```
$ sort temp1 temp2 temp3
$ sort	//sorts its standard input
```
### pipe
- a way to connect the output of one program to the input of another program without any temporary file
- a *pipeline* is a connection of two or more programs through pipes
- the ==vertical bar character== **"|"** tells the shell to set up a pipeline
```
$ who | sort	//prints sorted list of users
$ who | wc -l	//counts users
$ ls | wc -l	//counts files
$ ls | pr -3	//3-column list of filenames
$ who | grep mary	//look for particular user
```
* any program that adheres to the **convention** can be used in pipelines
```
$ ls | pr -3 | 1pr	//create a 3-column list of filenames on the line printer
$ who | grep mary | wc -l	//counts how many times mary is logged in
```
* the porgrams in a pipeline actually run at the same time, that means those can be interactive
* the kernel looks after whatever scheduling and synchronization is needed to make it all work
* the shell arranges things when you ask for a pipe
* **command invocation** in normal
  *command	optional-arguments	optional-filenames*
* Error massages from commands have to be handle differently, or they might disappear into a file or down a pipe

![](https://lh3.googleusercontent.com/proxy/vimQBbYkSTT1bic4qRHg2KDVHGEUpLvfhiv3tFKWA-PVFB3JbT_-9DYxzWlzqioMSF4sglRfGGUudGjurauiyP_GAUzaYB0TOybFScENbhDz2WKMcXxVNAiPHJnLMlM)
### processes
- run two programs with one command line by separating the commands with a semicolon ==;==
```
$ date; who	//commands are executed in sequence
```
- when you don't want to wait for command to finish its work before you start something else
```
$ wc ch* >wc.out &
6944	//process-id printed by the shell
```
- the ampersand ==&== means "start this command running, then take further commands from the terminal immediately"
- you can use **process-id** in other commands to refer to a specific running program, if several programs are running at the same time, each is a separate process with a different process-id.
```
$ pr ch* | lpr &	//
6951	//process-id
```
- the ==&== applies to the whole pipeline, only one id is printed
```
$ wait		//wait until all processes initated with & have finished
$ ctrl-c	//interrupt wait with DELETE
$ kill 6944	//stop a process initated with &
```
- if you forget the id
```
$ ps	//tell you about everything you have running
$ kill 0	//kill all your processes except your login shell
$ ps -ag	//tell you about all processes that other users are doing
 PID TTY TIME CMD
 	blablabla
```
>PID: process-id
TTY: the terminal associated with the process(as in who)
TIME: the processor time used in minutes and seconds
CMD: the command being run

### Hierarchical structure(parent/children)
![The UNIX file structure](https://www.prismnet.com/~mcmahon/Unix_Tutorial/Images/file_structure1.png)
- as you run commands, those processes are the children of your shell
- if you run a program from within one of those,that creates its own child process which is thus a grandchild of the shell

### nohup
- no hangup: **continue to run the process if you log out**, any output from the command is saved in a file called nohup.out
```
$ nohup command &
```
### nice
- run your job with lower than normal priority(it's kind to those who share your system if your process take a lot of processor resources)
```
$ nice expensive-command &
```
### at
- tell the system when to start your process
```
$ at time
whatever commands you want
blablabla
ctrl-d
$
```
- commands could come from a file
```
$ at 3am <file
```
>time can be written in 24-hour style like 2130, or 12-hour style like 930pm

### Tailoring the environment
- change the default of the convention in the local computing environment
```
$ stty erase e kill k	//type e to erase, and k is for line kill
```
- but you have to type this every time you log in
- if there is a file named **.profile** in the login directory, the shell will execute the commands in it when you log in, before printing the first prompt
```
	stty erase backspace
	stty erase '^h'	//ctrl-h
```
>the **^** character is an obsolete synonym for the pipe operator **|**, so you must protect it with quotes
```
	stty erase '^h' -tabs	//if your terminal doesn't have sensible tab stops
	who | wc -l	//counts the users to see how busy the system is
```
- some properties of shell are controlled by *shell variables*, with values that you can access an set it yourself
```
	PS1='Yes dear?	//the prompt string is actually stored in a shell variable called PS1
```
>the quotes are necessary since there are spaces in the prompt string,spaces are not permitted around the **=** in this construction
```
	MAIL=/usr/mail/you
```
### PATH
**search path**: the sequence of directories where the shell looks for commands, first in the current directory, then in /bin, and then in /usr/bin
```
	PATH=.:/bin:/usr/bin:/usr/games
	PATH=$PATH:/usr/games	//same thing
```
>directories separated by colons, remember that **.** is the current directory(can be omit, same as a null component)
### $
- you can obtain the value of any shell variable by prefixing its name with a ==$==
```
	d=/horribly/long/directory/name	//add to .profile
$ cd $d	//you don't have to type the long path
```
- if you intend to use the variables in other programs
```
	export MAIL PATH TERM d
```
### TERM
- names the kind of terminal you are using
```
	TERM=adm3	//add to .profile
```
## Chapter 2 - The file system
### od
* **octal** dump
* prints a visible representation of all the bytes of a file
```
$ od -c junk
0000000 n o w   i s   t h e   t i m e \n
0000020 f o r   a l l   g o o d   p e o
0000040 p l e \n
0000044
$
```
> the **-c** option means "interpret bytes as charaters."(**-x** show hexadecimal)
> use **-b** option will show the bytes as octal numbers as well.
### special character
* backspace: \b
* tab: \t
* newline: \n
* carriage return: \r
> [carriage return(迴車) & newline(換行)](https://blog.csdn.net/xiaoxian8023/article/details/8448160)
### line-at-a-time input
* 
## Chapter 3

## Chapter 4
>There is a large family of unix programs that read some input, perform a simple transformation on it, and write some output.
Such programs are called **filters**.
### grep
- In chapter 1:
```
$ grep pattern filenames...
```
- *grep* has looked for ordinary strings of letters and numbers, but can actually search for much more complicated patterns(a slightly restricted form of the string -> regular expressions)
- originally created by straightforward surgery on *ed*
```
$ grep -n variable *.[ch]	//locate variable in C source
$ grep From $MAIL | grep -v mary	//print message headers in mailbox that didn't come from mary
$ grep -y mary $HOME/lib/phone-book 	//find mary's phone number
$ who | grep mary 	//see if mary is logged in
$ ls | grep -v temp		//filenames that don't contain temp
```
- *-n* : prints line numbers
- *-v* : inverts the sense of the test
- *-y* : makes lower case letters in the pattern match letters of either case in the file(upper case still matches only upper case)
- metacharacters **^** and **$** anchor the pattern to the **beginning** and **end** of the line ( such metacharacters overlap with shell, so it's a good idea to enclose *grep* patterns in ==single quotes== )
```
$ grep '^From' $MAIL	//prints lines that begin with "From"
```
- if a *grep* character class begins with a ==circumflex==, the pattern matches any character except those in the class.
> **[^0-9]** matches any non-digit
- In shell, a ==bachslash== protects **]** and **-** in a character class, but *grep* and *ed* require that these character appear where their meaning is **unambiguous**
> **[][-]** matches either an opening or closing square bracket or a minus sign
- A ==period== is equivalent to the shell's **?** (it matches any character)
```
$ ls -l | grep '^d'		//list subdirectory names
$ ls -l | grep '^.......rw'		//list files others can read and write
```
- The ==closure== operator applies to the previous character or metacharacter in the expression, they match any number od successive matches of the character or metacharacter
	- closure applies to only one character
	- "any number" includes zero(no any), so if you want at least one character to be matched, you must duplicate it
> **x\*** matches a sequence of x's as long as possible
> **[a-zA-Z]\*** matches an alphabetic string
> **.\*** matches anything up to a newline
> **.\*x** matches anything up to and including the last x on the line

> **xy\*** matches an x followed by y's
> **[a-zA-Z][a-zA-Z]\*** matches a letter followed by zero or more letters(>=0)

- This command searches for users without passwords:(the second field of the password file is the encrypted password, so the second field must be empty)
```
$ grep '^[^:]*::' /etc/passwd		//beginning of line, any number of non-colons followed by a double colon.
```

#### fgrep
- searches for many literal strings simultaneously
#### egrep
- interprets true regular expressions
- the same as grep, but with an "or" operator and parentheses to group expressions.

### tr

### dd

### uniq

### sort

### Data transformers

### Programmable filters

### sed(stream editor)
>A derivative of ed that takes a "program" of editor commands and streams data from the files past them, doing the commands of the program on evry line.
### awk
>It includes arithmetic, variables, built-in functions, and a programming language that looks quite a bit like C.

## Chapter 5

### cal
- calendar
```
$ cal 10 1983	//print out the October in year 1983
    October 1983
S  M  Tu  W  Th  F  S
	(bla)
	(bla)
	(bla)
	(bla)
	(bla)
$ cal 10	//print out entire year 10
```
- you can place a command in your private *bin* directory that converts a more convenient argument syntax into whatever the real *cal* requires
### case
```
	case word in
	pattern) commands;;
	pattern) commands;;
	...
	esac
```
- the ==case== statement compares word to the patterns form top to bottom, and performs the commands associated **with (only) the first pattern that matches**
>each action is terminated by the double semicolon ==;;==
```
$ cat cal
# cal: nicer interface to usr/bin/cal

case $# in	//check the number of arguments, choose the appropriate action
0)    set 'date'; m=$2; y=$6 ;;  # no args: use today
1)    m=$1; set 'date'; y=$6 ;;  # 1 args: use this year
*)    m=$1; y=$2 ;;  # 2 args: month and year
esac	//catch-all (execute if neither 0 or 1, must be last)
//set month and year
case $m in
jan*|Jan*)	m=1 ;;	//could also be written as[jJ]an*
feb*|Feb*)	m=2 ;;
mar*|Mar*)	m=3 ;;
blablabla
...
...
...
nov*|Nov*)	m=11 ;;
dec*|Dec*)	m=12 ;;
[1-9]|10|11|12) ;;	# numeric month
*)	y=$m; m="" ;;	# plain year
esac

/usr/bin/cal $m $y	# run the real one
$
```
- the shell variable **$#** holds **the number of arguments** that a shell file was called with


| Shell Built-in Variable |                                                           |
|:-----------------------:|:--------------------------------------------------------- |
|           $#            | the number of arguments                                   |
|           $*            | all arguments to shell                                    |
|           $@            | similar to $*, see section 5.7                            |
|           $-            | options supplied to the shell                             |
|           $?            | return value of the last command executed                 |
|           $$            | processed-id of the shell                                 |
|           $!            | processed-id of the last command started with &           |
|          $HOME          | default argument for cd command                           |
|          $IFS           | list of characters that separate words in arguments       |
|          $MAIL          | file that, when changed, triggers "you have mail" massage |
|          $PATH          | list of directories to search for commands                |
|          $PS1           | prompt string, default '$ '                               |
|          $PS2           | prompt string for continued command line, default '> '    |

### set
```
	set 'date'
```
- shows the values of variables in the environment
- recognizes several options, of which the most often used are -v and -x, they turn on echoing of commands as they are being processed by the shell.
#### different patterns matching rules
- *ed* use '.' and shell use '?' for match "any character"
(p.138)

### which

### loops

#### for
```
	for i in list_of_words
	do
		(loop body, $i set to successive elements of list)
	done
	
	for i	//list is implicitly all arguments to shell file
	do
		(loop body, $i set to successive arguments)
	done
```

#### while
```
	while command
	do
		(loop body executed as long as command returns true)
	done
```
#### until
```
	until command
	do
		(loop body executed as long as command returns false)
	done
```