# [AIdrifter CS 浮生筆錄](https://hackmd.io/s/rypeUnYSb)<br> Thread Synchronization in Linux

## Create Process & Wait Process
### 常見的Create and Wait
- Create child process
- `fork()` (<unistd.h>)
- `system()`
- exec – 有 `execve()` 等共六個 functions (<unistd.h>)

- Wait child process
- `wait()` & `waidpid()` (<sys/types.h> <sys/wait.h>)

- Wait **other** processes
- 建議 **popen** + **ps** + **grep**
- **popen ??**
- Third-party library -> libprocps
### fork() Theorem
- Unix/Linux 以 `fork()` 產生 process，`exec()` 執行 process
- 呼叫 `fork()` 的為 Parent，產生出的為 Child，運行先後順序無法保證，用 `vfork()` 可以保證 Child 一定先運行 (直到exit 或 exec)
- Child 將完美複製整個 Parent，但由於**太浪費記憶體**，現有系統不再完整複製，只複製變更的部分 (**copy-on-write**)
- Process 結束後會產生結束狀態（程式碼，Exit code，CPU使用時間) 由系統保管，直到 Parent 領回才真正消滅
- 如果 Parent 先結束
- 把 Child 托孤給 **init**
- Parent 用 `wait()` 等待 Child
- Parent 先轉成 **idle**，直到 Child 結束，Parent 領回 Child 的結束狀態後繼續運行
- Parent 活著且不處理 Child 的結束狀態
- Child 變成 **zombie** 且 zombie 一直存在 (占用 Process List)
- Wiki **Hello World**
- On success, the PID of the child process is returned in the parent,
and 0 is returned in the child. On failure, -1 is returned in the
parent, no child process is created, and errno is set appropriately.

int main(void)
{
   pid_t pid = fork(); // parenet get child pid, child get 0

   if (pid == -1) {
      perror("fork failed");
      exit(EXIT_FAILURE);
   }
   else if (pid == 0) {
      printf("Hello from the child process!\n");
      _exit(EXIT_SUCCESS);
   }
   else {
      int status;
      (void)waitpid(pid, &status, 0); // parent wait child pid
   }
   return EXIT_SUCCESS;
}

### How to Prevent Zombine
- 有時候 Parent 正在忙，沒空處理 Child 的結束狀態
- 利用兩次 `fork()`，第一個 child 扮演原本的 parent，第二個 child 扮演真正的 child
- 原本 Parent 結束，兩個 Child 都托孤給 init -> 不會有 **zombie**

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include <unistd.h>
 
#define COUNT 3
 
int main()
{
    pid_t pid = 0;
    printf("parent process %u  \n",getpid());
    
    if ((pid = fork()) == 0)
    {
        int i = 0;
        while(i++ < COUNT)
        {
            printf("first child %u  pid %u \n",getpid(), pid);
            sleep(1);
        }
    }
    else
    {
        if ((pid = fork()) == 0)
        {
            int i = 0;
            while(i++ < COUNT)
            {
                printf("second child %u  pid %u \n",getpid(), pid);
                sleep(1);
            }
        }
        else
        {
            int i = 0;
            while(i++ < COUNT)
            {
                printf("parent died %u  pid %u \n",getpid(), pid);
                sleep(1);
            }
        }
    }
 
}

- output result

parent process 73802
parent died 73802  pid 73804
first child 73803  pid 0
second child 73804  pid 0
first child 73803  pid 0
parent died 73802  pid 73804
second child 73804  pid 0
parent died 73802  pid 73804
second child 73804  pid 0
first child 73803  pid 0

- gdb

(gdb) info proc
process 73130
cmdline = '/home/aidrifter/a.out'
cwd = '/home/aidrifter'
exe = '/home/aidrifter/a.out'

- 利用兩次 `fork()`，第一個 child 扮演原本的第二個child 的 parent，再托孤給 init

     if (fork() == 0) 
     {
          pritnf("first child %u \n",getpid());
          if (fork() == 0)
               pritnf("second child %u \n",getpid());
          else
               pritnf("first child died %u \n",getpid());
     }
     else 
     {
          // Parent 領回第一個 child 的結束狀態，
          // 再繼續做 Parent 該忙碌的事情 
     }

## Create Thread & Wait Thread
### Create Pthread and Wait

// sucess : return  0                                                                                                                                                                                              
// failed : return EAGAIN，EINVAL，EPERM
int pthread_create (               // 建立一個新的 thread
        pthread_t *tid,            // 指標指向 thread ID
        pthread_attr_t *attr,      // thread 屬性設定
        void *func(void *),        // function name
        void *arg);                // parameter for function
  
  
// sucess : return  0
// failed : return EDEADLK, EINVAL, ESRCH
int pthread_join (              // 等待某個 thread 结束
        pthread_t tid,          // thread ID
        void **retval);         // pthread_exit() 回傳值

### Hint
- 沒有特別設定，一個 thread 可以被 `pthread_join()` **等待**
- 不想被等待的話有兩種方式
- 建立前設定 `attr`
- `pthread_attr_init()` 做 init，
- `pthread_attr_setdetachstate()` 設定 detach
- `pthread_create()` 傳入 attr
- `pthread_attr_destroy()` 用完 attr 後要記得 free resource
- 建立完後才用 pthread_detach (thread_id);

- #include <pthread.h>
- gcc 的時候參數 `–lpthread`
- Reference
- [POSIX Threads Programming](https://computing.llnl.gov/tutorials/pthreads/)

### basic pthread
- 會先印出 num=110 幾秒過後再印出 x=100

void main(void) {
     int x = 100;
     pthread_t tid = 0;

     if ((pthread_create(&tid, NULL, aa, 
                      (void *)(&x))) != 0) 
    {
          printf (“Create thread failed \n”);
          return;
     }
     pthread_join (tid,NULL);
     printf (“x = %d\n”, x);
}

void *aa(void *arg) {
     int num = 0;
     
     num = *( (int *)arg );
     num = num + 10;
     printf (“num = %d\n”, num);
     sleep (5);
}

## Mutex
- 保證一次只有一個進入
### Mutex relation function in Linux

/* create one Mutex */
int pthread_mutex_init (
        pthread_mutex_t *mutex,          //  Init mutex
        pthread_mutexattr_t *attr);     // set NULL commonly

/* lock mutex */
int pthread_mutex_lock (
        pthread_mutex_t *mutex );

/* release mutex */
int pthread_mutex_unlock (
        pthread_mutex_t *mutex );

/* try to lock mutex(no wait) */
int pthread_mutex_trylock (
        pthread_mutex_t *mutex );

/*
 * release mutex back to OS
 * sucess: return 0
 * failed: Error num > 0
 *
 */
int pthread_mutex_destroy(
        pthread_mutex_t *mutex );

### Concept : Mutex in Linux（attr）
- 用法跟 Windows 差不多，都是在**保證一次只有一個thread能夠進入**
- 正常狀況下，pthread_mutex_lock() 不允許重複進入 (跟 Window 不同)，重複進入將造成 deadlock
- 想要 **Recursive** 时候**repeatly 進入 Mutex**
- 利用 **attr** 設定 `PTHREAD_MUTEX_RECURSIVE`
- 一樣，mutex_lock 幾次，就要相對 mutex_unlock 幾次
- How to use attr in pthread ?

// Initialize attr
pthread_mutexattr_t  attr;
pthread_mutexattr_init (&attr); 
pthread_mutexattr_settype (&attr, PTHREAD_MUTEX_RECURSIVE); // set attr

// create Mutex 的時候傳入 attr
pthread_mutex_init (&g_mutex, &attr); 

// release attr
pthread_mutexattr_destroy(&attr);  

- pthread mutex sample code
- mutex is in **user sapce**, some guy create shm to communicate with each thread(**cross-processes**)

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include <sys/types.h>
#include <pthread.h>
#include <unistd.h>
#include <sys/syscall.h>
#define gettid() syscall(__NR_gettid)

pthread_mutex_t  g_mutex = PTHREAD_MUTEX_INITIALIZER;
int j = 0;

int aa(void)
{
    pthread_mutex_lock (&g_mutex);
    j += 2;
    printf("j = %u tid %lu \n",j ,gettid());
    pthread_mutex_unlock (&g_mutex);
    return 0;
}

int bb(void)
{
    pthread_mutex_lock (&g_mutex);
    j += 3;
    printf("j = %u tid %lu\n",j ,gettid());
    pthread_mutex_unlock (&g_mutex);
    return 0;
}

int main(void)
{
    pthread_t tid1,tid2;
    pthread_mutex_init(&g_mutex,NULL);

    /* create aa & bb threads */
    pthread_create (&tid1, NULL, aa, NULL);
    pthread_create (&tid2, NULL, bb, NULL);

    /* wait aa bb threads */
    pthread_join (tid1, NULL);
    pthread_join (tid2, NULL);
    printf("j = %u tid %lu\n",j ,gettid());

    pthread_mutex_destroy(&g_mutex);
    return 0;

}

- gcc compile parameter

gcc thread.c -lpthread

### Ignore pthread_mutex_init()
- [PTHREAD_MUTEX_INITIALIZER vs pthread_mutex_init ( &mutex, param)](https://stackoverflow.com/questions/14320041/pthread-mutex-initializer-vs-pthread-mutex-init-mutex-param)

pthread_mutex_t g_mutex = PTHREAD_MUTEX_INITIALIZER;
// lock
pthread_mutex_lock (&g_mutex);
{
    /* Critical Section (memory operation)*/
}
// unlock
pthread_mutex_unlock (&g_mutex);

## Shared-Exclusive Lock
- 可以同時讀取，不能同時寫入，
也不能同時讀取+寫入
(限定同一個 process 的 threads 共同使用)
- 用在有些 threads 讀取共用資源，有些 threads 修改
- 幾個 threads 可以同時讀取，不會有問題
- 讀取與寫入不可同時發生，靠鎖定保證沒有這樣的狀況

## File Lock
- 類似 Shared-Exclusive Lock
可以跨 process 使用

- 利用 fcntl() 實現，fcntl() 功能強大，這只邊介紹 File Lock
- 利用一個檔案本來就可**同時被讀取**，但不能同時**讀取+寫入**
- 需要 `<unistd.h>` , `<fcntl.h>`
- 通常可以用來
- Multi-process 同時使用一個資源狀況下，做到 Read/Write Lock
- 利用 File Lock 性質，在 Multi-process 建立 Critical Section
- 可以當成【如果 Process A 運行中，Process B 不運行】，舉例來說可以做成 Singleton (確保只有一個相同的 Daemon 在運行)

- CMD Parameter
- **F_SETLK**：執行鎖定，鎖定方式由 `stLock` 指定，若無法鎖定將返回EACCES 或 EAGAIN，不等待
- **F_SETLKW**：無法鎖定時會等待，直到等到為止
- **F_GETLK**：取得目前鎖定狀況，結果存放在`stLock`中
- `stLock`.**l_type** 可以是
- F_RDLCK (Read Lock)
- F_WRLCK (Write Lock)
- F_UNLCK (Unlock)
- `stLock` 的 **l_whence**, **l_start**, **l_len** 可指定鎖定 File 的某個範圍

#include <unistd.h>                                                                                                                                                                                                
#include <fcntl.h>
  
/*
 * return value
 * sucess： return value accronding to cmd
 * failed： -1
 *
 */
extern int fcntl (
        int fd,                  // use open() to get File descriptor
        int cmd,                 // F_SETLK, F_SETLKW, F_GETLK
        struct flock *stLock);   // flock structure
                                                           

## Event
### Condition Variable 相關 functions

// create Condition Variable
int pthread_cond_init (
        pthread_cond_t *cond,        // Create Condition Variable
        pthread_condattr_t *attr );  // 建立的设定，可以给 NULL 就好

// wait some Condition Variable
int pthread_cond_wait (
        pthread_cond_t *cond,       // Wait Condition Variable
        pthread_mutex_t *mutex);    // Accompany wait 所使用的 mutex

// 通知一個wait的 thread
int pthread_cond_signal (
        pthread_cond_t *cond);

 // 通知所有wait的 threads
int pthread_cond_broadcast (
        pthread_cond_t *cond);

// release condition
// sucess : return 0
// failed : Error num > 0
int pthread_cond_destroy (pthread_cond_t *cond);

### Condition Variable Usage

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

pthread_cont_t g_cond;
pthread_mutex_t g_mutex;

void *aa(void *arg) 
{
     …
     /* 做完, signal 通知 (event)*/
     pthread_cond_signal (&g_cond,             
                          &g_mutex);
                          
}

main() 
{
     pthread_t tid;
     // init condition
     pthread_cond_init(&g_cond, NULL);
     pthread_mutex_init(&g_mutex, NULL);
     
     // create thread aa
     pthread_create(&tid, NULL, aa, NULL);
     …
     
     // wiat until aa terminates 
     pthread_cond_wait(&g_cond, &g_mutex);  
    
     // aa has been finished, we can continue ...
     …

     // release source
     pthread_cond_destroy(&g_cond);
     pthread_mutex_destroy(&g_mutex);
}

### Signal Function

// clear set
int sigemptyset (sigset_t *set)

// add one signal to set group
int sigaddset (
            sigset_t *set,   // signal set
            int signum);     // 要加入的 signal number

// delete one signal form signal set
int sigdelset (
        sigset_t *set,      // signal set
        int signum);        // delete signal number

// set waitting singnals
int sigprocmask (
        int how,                 // wait way : SIG_BLOCK, SIG_UNBLOCK, SIG_SETMASK
        const sigset_t *set,     // 打算等待的哪些 signals (signal set)
        sigset_t *oldset);       // 返回原本的 set，可以是 NULL

// wait set 裡面的all signal(s)
int sigwait (*set, int* sig);   

// notice some process ->  signum has already happened
int kill (pid, signum);

### Use Signal Like Event

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <pthread.h>
#include <sys/types.h>
#include <unistd.h>

void *aa(void *arg)
{
    pid_t ppid=0;

    // Get parent pid
    ppid = getppid();
    printf(" %s is working \n",__func__);

    // aa finished, notice via signal
    kill (ppid, SIGUSR1);
}

int main() {
    pthread_t tid;  sigset_t set;
    int sig=0;

    // Init set (only wait SIGUSR1)
    sigemptyset (&set);
    sigaddset (&set, SIGUSR1);
    sigprocmask (SIG_SETMASK, &set, NULL);

    // Create thread aa
    pthread_create (&tid, NULL, aa, NULL);



    // Wait aa's signal terminate (SIGUSR1)
    sigwait (&set, &sig);
    // aa was already finished , continue ...
    // …
    printf(" %s is working \n",__func__);

    return 0;
}

- [FIXME] prcoess will terminate tmux when `kill()`

(gdb) bt
#0  kill () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555554952 in aa (arg=0x0) at event.c:17
#2  0x00007ffff7bc06da in start_thread (arg=0x7ffff77f1700) at pthread_create.c:456
#3  0x00007ffff78fad7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
(gdb) c
Continuing.
User defined signal 1

## Semaphore
### Semaphore Function

// Create one anonymous Semaphore
// sucess: return 0
// failed: return -1
int sem_init (
        sem_t *sem,             // 要使用的 Semaphore
        int pshared,            // 0: 不能跨 proc 使用，非0: 可以跨 proc
        unsigned int value );   // 一開始的 resource count

// Create one named Semaphore
// sucess: return 0
// failed: return -1
sem_t sem_open (
        const char *name,       // 取名字，透過名字共用
        int oflag,              // O_CREAT，O_CREAT | O_EXCL
        mode_t mode,            // S_IRWXU，S_IRUSR…，S_IRWXG,  S_IWGRP…，S_IRWXO，S_IXOTH…
        unsigned int value );   //一開始的 resource count

int sem_post (sem_t *sem);     // increase resource. unlimited , never sleep
int sem_wait (sem_t *sem);     // decrease resource，sleep when  resource count = 0
int sem_trywait (sem_t *sem);  // decrease resource，Don't slee[

int sem_getvalue (sem_t *sem, int *sval); // 取得目前 count (放 sval)

int sem_close (sem_t *sem);     // close sem_t from sem_open()
int sem_destroy (sem_t *sem);   // destory sem_t from  sem_init()
int sem_unlink (sem_t *sem);    // release sem_t's resource from sem_open()

### Semaphore Concept
- 使用 `sem_wait()` or `sem_trywait()` 當消費者
- 消費者一次把 Resource count 減一
- Resource count=0 的時候，消費者睡覺

- 用 `sem_post()` 當生產者
- 生產者不會真的睡覺
- 連上限都沒有
- 生產者要睡覺，要自己想辦法

- 單純使用`sem_init()` + `fork()`，Parent 跟 Child 將不會共用同一個semaphore，要加上 shared memory 配合使用。 ==這是個常見的陷阱==