---
# System prepended metadata

title: 11/25 開會
tags: [AICA]

---

# 11/25 開會 
###### tags: `AICA`

> [會議連結（21：30～）](https://teams.microsoft.com/l/meetup-join/19:meeting_NzlhMTMwZDUtNzU4OC00Mjg3LTk0YzQtMTVmNWE1NGIzZjk1@thread.v2/0?context=%7B%22Tid%22:%22c2e7753f-aa05-4abc-8c02-293ad122ca19%22,%22Oid%22:%22fdf9c304-4ad1-4191-955a-b0e10b7fcf43%22%7D)

---

## Mediapipe Hands
- [安裝方式](https://aijishu.com/a/1060000000182238)
- [mediapipe visualizer](https://viz.mediapipe.dev/)
    - 視覺化方法了解mediapipe

- [手部提取模型](https://github.com/google/mediapipe)
- [利用圖片辨識 + 範例code](https://blog.csdn.net/stq054188/article/details/114646071)
    - 動態中擷取圖片就可實作
- 實作方法
    - 利用21個節點
        - 困難：確認角度
        - (目前還看不太懂角度的辨別
    

## Pygame
- [Pygame基本介紹](https://hackmd.io/@NetJagaimo/B1scMoNxS)
    - 設定視窗
    - 畫幾何圖形
    - 基本架構
    - 輸入: 鍵盤，滑鼠
    - 輸出: 影像，音樂

### GUI
- method1
:bulb: [PyQt5](https://zhung.com.tw/article/pyqt%E5%85%A5%E9%96%80%E7%94%A8python%E5%AF%AB%E7%AC%AC%E4%B8%80%E6%94%AFgui/)
:bulb: [PyQT5速成教學之Qt Designer介紹與入門](https://www.it145.com/9/49744.html)
:bulb: [詳解Python GUI程式設計之PyQt5入門到實戰](https://www.796t.com/article.php?id=179975)
    - 可結合python,但能不能結合pygame眾說紛紜，好像可以不用到pygame(?
    - 版面設計不難，難的是如何控制按鍵要做什麼功能
    - 網路上都說較難上手
    
    
- method2
:bulb: [Tkinter](https://blog.techbridge.cc/2019/09/21/how-to-use-python-tkinter-to-make-gui-app-tutorial/)
    - python內建的GUI Programming
    - 但版面好像要自己coding
    - [Pygame和tkinter的結合-畫圓](https://www.796t.com/post/N2d3cjg=.html)
```python=
    embed = tk.Frame(root, width = 500, height = 500) 
    #creates embed frame for pygame window
```
    
- method3
    - 純粹靠pygame疊圖，圖的座標自行設定
    - 應用pygame中滑鼠點擊事件來觸發

## 整合

==1. 把 Midiapipe Hands & Pygame 安裝到樹莓派上的方法==

### **Mediapipe**

![](https://i.imgur.com/g8OzwTK.jpg)

#### [Mediapipe Hands on RPi](https://circuitdigest.com/microcontroller-projects/gesture-controlled-media-player-using-raspberry-pi-and-mediapipe)
- 現成的安裝code + 一個控制影片的範例
- 可參考的做法
    - 手指數量
    - 21 個點的位置座標
- ==用這個的話是不是不用再自己 train 模型了? 所以就沒有模型需要統一 format 的問題?==

#### [在樹莓派上使用MediaPipe框架](https://medium.com/@RouYunPan/%E5%9C%A8%E6%A8%B9%E8%8E%93%E6%B4%BE%E4%B8%8A%E4%BD%BF%E7%94%A8mediapipe%E6%A1%86%E6%9E%B6-fa766a243a08)
- MediaPipe框架安裝要求
    - 電腦: x86/arm64架構 , OpenCV 3.x, Python 3.7.
    - 樹梅派: [Ubuntu](colab.research.google.com)、[docker](https://phoenixnap.com/kb/docker-on-raspberry-pi)


#### API介紹與範例
1. STATIC_IMAGE_MODE
有false跟true兩種模式，其中false適合用來連續動態偵測，此模式會把第一張讀到的照片的手當標準，持續追蹤。
2. MAX_NUM_HANDS
要檢測的最大手數，一般設2
3. MIN_DETECTION_CONFIDENCE
成功檢測為手的信心程度

```python=
import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles
mp_hands = mp.solutions.hands

# For static images:
IMAGE_FILES = []
with mp_hands.Hands(
    static_image_mode=True,
    max_num_hands=2,
    min_detection_confidence=0.5) as hands:
  for idx, file in enumerate(IMAGE_FILES):
    # Read an image, flip it around y-axis for correct handedness output (see
    # above).
    image = cv2.flip(cv2.imread(file), 1)
    # Convert the BGR image to RGB before processing.
    results = hands.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

    # Print handedness and draw hand landmarks on the image.
    print('Handedness:', results.multi_handedness)
    if not results.multi_hand_landmarks:
      continue
    image_height, image_width, _ = image.shape
    annotated_image = image.copy()
    for hand_landmarks in results.multi_hand_landmarks:
      print('hand_landmarks:', hand_landmarks)
      print(
          f'Index finger tip coordinates: (',
          f'{hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP].x * image_width}, '
          f'{hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP].y * image_height})'
      )
      mp_drawing.draw_landmarks(
          annotated_image,
          hand_landmarks,
          mp_hands.HAND_CONNECTIONS,
          mp_drawing_styles.get_default_hand_landmarks_style(),
          mp_drawing_styles.get_default_hand_connections_style())
    cv2.imwrite(
        '/tmp/annotated_image' + str(idx) + '.png', cv2.flip(annotated_image, 1))
    # Draw hand world landmarks.
    if not results.multi_hand_world_landmarks:
      continue
    for hand_world_landmarks in results.multi_hand_world_landmarks:
      mp_drawing.plot_landmarks(
        hand_world_landmarks, mp_hands.HAND_CONNECTIONS, azimuth=5)

# For webcam input:
cap = cv2.VideoCapture(0)
with mp_hands.Hands(
    model_complexity=0,
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5) as hands:
  while cap.isOpened():
    success, image = cap.read()
    if not success:
      print("Ignoring empty camera frame.")
      # If loading a video, use 'break' instead of 'continue'.
      continue

    # To improve performance, optionally mark the image as not writeable to
    # pass by reference.
    image.flags.writeable = False
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    results = hands.process(image)

    # Draw the hand annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_hand_landmarks:
      for hand_landmarks in results.multi_hand_landmarks:
        mp_drawing.draw_landmarks(
            image,
            hand_landmarks,
            mp_hands.HAND_CONNECTIONS,
            mp_drawing_styles.get_default_hand_landmarks_style(),
            mp_drawing_styles.get_default_hand_connections_style())
    # Flip the image horizontally for a selfie-view display.
    cv2.imshow('MediaPipe Hands', cv2.flip(image, 1))
    if cv2.waitKey(5) & 0xFF == 27:
      break
cap.release()
```

### **Pygame**

#### Pygame on RPi
- 安裝 (不確定是哪一個)
```python=
sudo apt-get install python-pygame
```
```python=
sudo apt-get install python3-pygame
```
- 如果沒有理解錯誤的話，安裝完好像就可以直接在樹莓派執行我們寫好的 pygame 的 .py 檔
```python=
python xxxxxx.py
```

==2. 怎麼把 Mediapipe Hands 的結果傳到 Pygame 的方法==

### **Method 1 - Keyboard's Keypress**
:bulb: [code 參考連結](https://www.analyticsvidhya.com/blog/2021/06/gesture-controlled-video-game/)

- 比較簡單，但不確定在樹莓派上能不能成功的做法
- 兩個 py 檔

mediapipe_hands.py
```python=
if 手勢辨識正確:
    keyboard.press(Key.right)
```

pygame.py
```python=
class Game:
    
    def xxxxxx:
    
    def oooooo:
    
    def correct(self):
        for event in pygame.event.get():
            if (event.key == pygame.K_RIGHT):
                答對的手勢亮起來
```

### **Method 2 - 用一堆待研究的 Mediapipe Hands 內建功能直接實現**

:bulb: [手部控制俄羅斯方塊 - 成果影片](https://youtu.be/sFDt9upueRE)
:bulb: [手部控制俄羅斯方塊 - Source Code](https://gist.github.com/kpkpkps/79357982a6044553baf3610ad39d0c90)

- Source Code 好像幾乎把我們要做的東西都包了
    - 遊戲介面
    - 手勢視窗
    - 音樂
    - Game Over
    - 計分
    - (大概只缺計時)
- 比較難，努力研究中，暫時還沒看懂
- 一個 py 檔

Source Code 裡 Mediapipe Hands 和 Pygame 的整合部分

```python=
#Set up the hand tracker
        success, img = cam.read()
        imgg = cv2.flip(img, 1)
        imgRGB = cv2.cvtColor(imgg, cv2.COLOR_BGR2RGB)
        results = hands.process(imgRGB)

        if results.multi_hand_landmarks:
            for handLms in results.multi_hand_landmarks:
                for id, lm in enumerate(handLms.landmark):
                    h, w, c = imgg.shape
                    if id == 0:
                        x = []
                        y = []
                    x.append(int((lm.x) * w))
                    y.append(int((1 - lm.y) * h))

                    #This will track the hand gestures
                    if len(y) > 20:
                        if (x[0] > x[3] > x[4]) and not(y[20] > y[17]):
                           left_wait += 1
                        if not(x[0] > x[3] > x[4]) and (y[20] > y[17]):
                            right_wait += 1
                        if (x[0] > x[3] > x[4]) and (y[20] > y[17]):
                            rotate_wait += 1


                mpDraw.draw_landmarks(imgg, handLms, mpHands.HAND_CONNECTIONS)

        else:
            down_wait += 1
```

## 有照流程拆的 Project 分工?






    
