# 隱形滑鼠開發日誌
- [name=Yu-Cheng Lin]
- [time=Sun, Jun 2, 2023 11:50 AM]
:::info
老兄早安，這裡是老兄們的隱形滑鼠開發園地，有加寫東西記得署名
:::
老兄先把報告要用的資訊打上來了，反正到時候會忘就像：
```
import brain
if not 先寫下來：
brain.forget(報告要用的)
return 蛤
```
### 研究動機
因為電腦教室的軌跡球~~難用~~使用上有些不順，帶滑鼠又頗麻煩，我們決定帶個程式來當滑鼠
### mediapipe
手部辨識節點編號
## 6/2 隱形滑鼠開發 Day1
基本上拍得到手，接下來要搞懂如何取得 landmarklist的 x y座標
https://hackmd.io/m7Y3tO3KRUyb2GfARoBHWA#
:::spoiler
```py=
import mediapipe as mp
import cv2
import pyautogui as pg
Cam = cv2.VideoCapture(0)
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles
with mp_hands.Hands(
model_complexity=0,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = Cam.read()
if not rec:
print('BRUH')
break
img = cv2.resize(img, (800,600))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
results = hands.process(img2)
if results.multi_hand_landmarks:
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
pg.moveTo(hand_landmarks)
cv2.imshow('BRUH', img)
if cv2.waitKey(5)==ord('q'):
break
Cam.release()
cv2.destroyAllWindows()
```
:::
今天不錯，上半天，放假囉
*by ELO*
---
## 6/3 Day2
#### 22:32
老兄，可以動了
不過這個跟我們真的要的不太一樣，我設定成根據食指指尖座標移動至螢幕的對應位置
這樣設定會有兩個問題：
1. 手移動到邊邊時，模型偵測不到，進而無法移動鼠標到底邊
2. 用起來挺智障的，要一直注意相機拍得到的區域，使用上不自然
#### 22:40
老兄，可以點擊了，有請見證人***正清則***
點擊的判斷目前是用拇指、食指、中指的指尖連線夾角cos值判斷，小於-0.7就判定點擊
另外我搞了一個新文件叫[**click_determine**](https://hackmd.io/m7Y3tO3KRUyb2GfARoBHWA?both#click_determine)，用來放點擊判斷的相關自訂義函式，老兄放在後面
現在的問題是：
1. 用cosθ<-0.7 (in click_determine, line28) 似乎不是一個精準的判斷，有時按不下去
2. 在點擊時，鼠標會因為手部動作而跟著移動，導致無法精準點擊
### main
:::spoiler
```py=
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from math import *
from click_determine import distance_counting, click, count_cos
Cam = cv2.VideoCapture(1) # 取用相機
# print(ag.position())
# print(ag.size())
Width, Height = ag.size() # 取得螢幕尺寸
print(Width, Height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=0,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = Cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800,600))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
for idx, hand_handedness in enumerate(results.multi_handedness):
# print(hand_handedness.classification)
# print(hand_handedness.classification[0].label)
WhichHand = hand_handedness.classification[0].label
if WhichHand == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
destinationX = (1-Hand_Landmark.landmark[9].x)*Width # 9是中指指根
destinationY = (1-Hand_Landmark.landmark[9].y)*Height
index_fingerX = Hand_Landmark.landmark[8].x # 8是食指指尖
index_fingerY = Hand_Landmark.landmark[8].y
thumb_tipX = Hand_Landmark.landmark[4].x # 4是大拇指指尖
thumb_tipY = Hand_Landmark.landmark[4].y
middle_fingerX = Hand_Landmark.landmark[12].x # 12是中指指尖
middle_fingerY = Hand_Landmark.landmark[12].y
ring_fingerX = Hand_Landmark.landmark[16].x # 16是無名指指尖
ring_fingerY = Hand_Landmark.landmark[16].y
pinkyX = Hand_Landmark.landmark[20].x # 20是小拇指指尖
pinkyY = Hand_Landmark.landmark[20].y
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [index_fingerX, index_fingerY]
thumb = [thumb_tipX, thumb_tipY]
ring_finger = [ring_fingerX, ring_fingerY]
''' # 顯示三指尖夾角cos值，試驗用
print(count_cos(distance_counting(index_finger, ring_finger),
distance_counting(index_finger, thumb),
distance_counting(ring_finger, thumb)))'''
ag.moveTo(destinationX, destinationY) # 移動鼠標
# 進行點擊的判斷
if click(index_finger, thumb, ring_finger):
ag.leftClick(destinationX, destinationY)
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
Cam.release()
cv2.destroyAllWindows
```
:::
### click_determine
:::spoiler
```py=
from math import *
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三邊長算cos用的
def count_cos(a, b, c):
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的主要程式
def click(tip1, tip2, tip3):
'''
:param tip1: location of index finger
:param tip2: location of thumb
:param tip3: location of ring finger
:return:
'''
a = distance_counting(tip1, tip2)
b = distance_counting(tip1, tip3)
c = distance_counting(tip2, tip3)
cos_theta = count_cos(a, b, c)
if cos_theta < -0.7:
return 1
```
:::
### 問題&討論
1. 是不是好像要改成根據手的相對移動來滑動鼠標？還是這樣是我們要的？
2. 以手指動作來判斷點擊的方法要思考一下，有人用手指三節點的角度判斷，我想試試看用五指尖節點相對位置來判斷，說不定還可以增加其他手指的快捷鍵
3. 來不來得及多做一點變化，畢竟這個已經有人做過了
*by ELO*
---
## 6/4 Day3
#### 13:28
參考了一下別人做的，老兄覺得要改一下程式結構，是時候引進class了(幹
#### 15:15
老兄剛剛解決了無法移動到邊緣的問題，把鏡頭畫面調大就好了(in main, line30)
不對，根本沒幫助，應該是要調整靈敏度
可以，我成功了
假如想要在座標(a,b)(a<b)之間就可以移動到整個螢幕，也就是(a,b)->(1,0)，可以將讀入的座標套入以下公式
:::success
**x'=b/(b-a)-(x/(b-a))\*width
y'=b/(b-a)-(y/(b-a))\*height**
:::
以程式碼為例，目前我是設定讓移動範圍在座標(0.2,0.7)之間，所以就變成
:::success
**destinationX = ((7/5)-(Hand_Landmark.landmark[9].x/0.5))\*Width
destinationY = ((7/5)-(Hand_Landmark.landmark[9].y/0.5))\*Height**
:::
新問題
目前的移動模式有個缺點，鼠標是沒辦法固定的，當手部被偵測到時數表就會一直動，沒辦法把鼠標放在一個位置，想想在看影片時首要一直收起來
### main
:::spoiler
```py=
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from math import *
from click_determine import *
Cam = cv2.VideoCapture(1) # 取用相機
# print(ag.position())
# print(ag.size())
Width, Height = ag.size() # 取得螢幕尺寸
print(Width, Height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = Cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
# for idx, hand_handedness in enumerate(results.multi_handedness):
# WhichHand = hand_handedness.classification[0].label
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
print(Hand_Landmark.landmark[9])
destinationX = ((7/5)-(Hand_Landmark.landmark[9].x/0.5))*Width # 9是中指指根
destinationY = ((7/5)-(Hand_Landmark.landmark[9].y/0.5))*Height # 若鏡頭對桌面的話，前面要1-才會上下相反
'''index_finger_tipX = Hand_Landmark.landmark[8].x # 8是食指指尖
index_finger_tipY = Hand_Landmark.landmark[8].y
thumb_tipX = Hand_Landmark.landmark[4].x # 4是大拇指指尖
thumb_tipY = Hand_Landmark.landmark[4].y
middle_finger_tipX = Hand_Landmark.landmark[12].x # 12是中指指尖
middle_finger_tipY = Hand_Landmark.landmark[12].y
ring_finger_tipX = Hand_Landmark.landmark[16].x # 16是無名指指尖
ring_finger_tipY = Hand_Landmark.landmark[16].y
pinky_tipX = Hand_Landmark.landmark[20].x # 20是小拇指指尖
pinky_tipY = Hand_Landmark.landmark[20].y'''
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
thumb = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
ring_finger = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
''' # 顯示三指尖夾角cos值，試驗用
print(count_cos(distance_counting(index_finger, ring_finger),
distance_counting(index_finger, thumb),
distance_counting(ring_finger, thumb)))'''
ag.moveTo(destinationX, destinationY) # 移動鼠標
# 進行點擊的判斷
# if click(index_finger, thumb, ring_finger):
# ag.leftClick(destinationX, destinationY)
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
Cam.release()
cv2.destroyAllWindows
```
:::
### click_determine
:::spoiler
```py=
from math import *
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三邊長算cos用的
def count_cos(a, b, c):
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的主要程式
def click(tip1, tip2, tip3):
'''
:param tip1: location of index finger
:param tip2: location of thumb
:param tip3: location of ring finger
:return:
'''
a = distance_counting(tip1, tip2)
b = distance_counting(tip1, tip3)
c = distance_counting(tip2, tip3)
cos_theta = count_cos(a, b, c)
if cos_theta < -0.7:
return 1
# 判斷左右手(限右手操控)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
```
:::
老兄沒有鏡頭可以試試webcam
*by ELO*
---
## 6/5 Day4
老兄今天小累，改一點結構就好
不行，我剛剛有個想法，如果用大拇指當左鍵觸發呢？大拇指的角度看起來頗適合
#### 21:56
<font color ="#FF1212">**不得了了各位，老兄剛剛發現大拇指超適合**</font>
拇指點擊比食指的判斷精準多一百倍，而且不太會有重複觸發的問題，甚至不會影響到鼠標的位置
不過我認為還是需要把兩個子執行緒想辦法分開，像[文獻2](https://blog.csdn.net/Already8888/article/details/126125467)一樣，可以在點擊暫停移動會更好
### main
:::spoiler
```py=
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from math import *
from click_determine import *
import threading
import time
Cam = cv2.VideoCapture(1) # 取用相機
# print(ag.position())
# print(ag.size())
Width, Height = ag.size() # 取得螢幕尺寸
print(Width, Height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = Cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
# for idx, hand_handedness in enumerate(results.multi_handedness):
# WhichHand = hand_handedness.classification[0].label
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark.landmark[9])
destinationX = ((7/5)-(Hand_Landmark.landmark[9].x/0.5))*Width # 9是中指指根
destinationY = ((7/5)-(Hand_Landmark.landmark[9].y/0.5))*Height # 若鏡頭對桌面的話，前面要1-才會上下相反
'''index_finger_tipX = Hand_Landmark.landmark[8].x # 8是食指指尖
index_finger_tipY = Hand_Landmark.landmark[8].y
thumb_tipX = Hand_Landmark.landmark[4].x # 4是大拇指指尖
thumb_tipY = Hand_Landmark.landmark[4].y
middle_finger_tipX = Hand_Landmark.landmark[12].x # 12是中指指尖
middle_finger_tipY = Hand_Landmark.landmark[12].y
ring_finger_tipX = Hand_Landmark.landmark[16].x # 16是無名指指尖
ring_finger_tipY = Hand_Landmark.landmark[16].y
pinky_tipX = Hand_Landmark.landmark[20].x # 20是小拇指指尖
pinky_tipY = Hand_Landmark.landmark[20].y'''
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
ring_finger = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
# 拇指指節，拇指指根，嘗試判斷點擊
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
'''# 顯示三指尖夾角cos值，試驗用
print(count_cos(distance_counting(thumb_ip, thumb_mcp),
distance_counting(thumb_ip, thumb_tip),
distance_counting(thumb_mcp, thumb_tip)))'''
ag.moveTo(destinationX, destinationY) # 移動鼠標
# 進行點擊的判斷
# if click(index_finger, thumb_tip, ring_finger):
# ag.leftClick(destinationX, destinationY)
# 點擊判斷2.0-拇指點擊版本(so far so good)
if click(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destinationX, destinationY)
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
Cam.release()
cv2.destroyAllWindows
```
:::
### click_determine
:::spoiler
```py=
from math import *
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三邊長算cos用的
def count_cos(a, b, c):
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的主要程式
def click(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return:
"""
a = distance_counting(point1, point2)
b = distance_counting(point1, point3)
c = distance_counting(point2, point3)
cos_theta = count_cos(a, b, c)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
```
:::
好，有進度就好
*by ELO*
---
## 6/6 Day5
今天我真的沒做事，老哥幫我們加入了靈敏度的設定，我不確定手指會不會成功保留在偵測範圍內，總之明天試
躺平
喔對了，老兄還有想到，如果改成點擊時暫停移動的話，會無法拖曳，所以這似乎不是個好方法，或是再尋求其他方式達成拖曳功能
*by ELO*
---
## 6/7 Day6
老兄剛剛想到昨天吃飯的時候，跟老哥提到把滑鼠設計成像xbox體感遊戲的操控，就是鏡頭直接水平拍，但是老哥說已經懶得帶滑鼠的人應該不太樂意用電腦還要手舉著
嗯，合理
---
## 6/9 Day7
#### 10:29
昨天沒做事，今天上課先來搞一下
關於鼠標很抖，嘗試改變鏡頭捕捉幀數來精準化手部偵測的位置
鄭清澤提出的sensitivity無法解決我們需要解決的問題，簡單來說，老兄搞錯重點
但是鄭清則同志替我們找到了autogui中的PAUSE，解決了移動次數不夠密集，也就是鼠標的移動不夠絲滑的問題，感謝您，同志
今天修改的部分有：
1. 鼠標移動卡頓問題（ag.PAUSE)
2. 移動到頂端時手指看得到了
3. 增加移動閾值(threshold)來避免手在不移動的情況下亂晃的問題
4. 主程式改為定義事件，main只負責callout
今天的待解問題：
1. 雖然鼠標不移動時可固定，但移動時仍會晃動
2. 滾輪、右鍵、側鍵功能（特別是滾輪，屌打軌跡球關鍵）
3. 左鍵回到食指操控
勁哥的點子：滾輪可以用類似觸控板的雙指驅動
### main
:::spoiler
```py=
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from math import *
from definitions import *
import threading
import time
if __name__ == '__main__':
hand_detect()
```
:::
### definitions
:::spoiler
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
import threading
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三邊長算cos用的
def count_cos(a, b, c):
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的主要程式
def click(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return:
"""
a = distance_counting(point1, point2)
b = distance_counting(point1, point3)
c = distance_counting(point2, point3)
cos_theta = count_cos(a, b, c)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 主程式
def hand_detect():
Cam = cv2.VideoCapture(0) # 取用相機
Cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
# print(ag.position())
# print(ag.size())
Width, Height = ag.size() # 取得螢幕尺寸
print(Width, Height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
prev_position = None
move_distance = 0
threshold = 7
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = Cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
# for idx, hand_handedness in enumerate(results.multi_handedness):
# WhichHand = hand_handedness.classification[0].label
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark.landmark[9])
destinationX = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * Width # 9是中指指根
destinationY = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * Height # 若鏡頭對桌面的話，前面要1-才會上下相反
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
ring_finger = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
# 拇指指節，拇指指根，嘗試判斷點擊
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
'''# 顯示三指尖夾角cos值，試驗用
print(count_cos(distance_counting(thumb_ip, thumb_mcp),
distance_counting(thumb_ip, thumb_tip),
distance_counting(thumb_mcp, thumb_tip)))'''
if prev_position is not None:
move_distance = distance_counting(prev_position, (destinationX, destinationY))
if move_distance > threshold:
ag.moveTo(destinationX, destinationY) # 移動鼠標
prev_position = (destinationX, destinationY)
# 進行點擊的判斷
# if click(index_finger, thumb_tip, ring_finger):
# ag.leftClick(destinationX, destinationY)
# 點擊判斷2.0-拇指點擊版本(so far so good)
if click(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destinationX, destinationY)
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
Cam.release()
cv2.destroyAllWindows
```
:::
---
## 6/10 Day8
#### 13:02
關於滾輪的部分老兄已經大概有想法了，鏡頭上中指的彎曲相對算明顯，所以我想嘗試直接判斷
老兄順便改了count_cos的內容
### 修改的defs
:::spoiler
```py=
from math import *
def wheel_determine(mid_point, side1_point, side2_point):
theta = count_cos(mid_point, side1_point, side2_point)
if theta > threshold: # 等滾輪測試
return 1
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
```
:::
---
## 6/11 Day9
#### 19:02
老兄今天凌晨看球，現在超累，不過老兄弄出滾輪了(in definitions, line139-153)
滾輪主要是中指指尖及第一二指節的角度判斷，平時夾角cos值約落在0.98(以我的鏡頭設置而言)
:::success
1. cosθ>-0.75 → 向下捲動(手指彎曲，夾角縮小)
2. cosθ<-0.995 → 向上捲動(手指伸直，角度趨近平角)
:::
目前每次偵測到的移動值是10(像素?不確定)，只要手指持續伸直或彎曲即可向上向下捲動
水啦，好東西
#### 今日問題：
1. 能用是能用，但判定依據也要再調整一下，不然手放著就開始飄，而且還要一直維持特定角度
2. 沒了，目前就這樣
或許可以改成角度改變才滾動，而不是到特定角度就滾動，但是再說，我要去吃飯
喔或是改用一二指節和指根的角度
### main
:::spoiler
```py=
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from math import *
from definitions import *
import threading
import time
if __name__ == '__main__':
hand_detect()
```
:::
### definitions
:::spoiler
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from definitions import *
import threading
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三座標算cos用的
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的程式
def click(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return:
"""
cos_theta = count_cos(point1, point2, point3)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
# 判斷偵測到的手(左右相反)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 主程式
def hand_detect():
cam = cv2.VideoCapture(1) # 取用相機
cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
# print(ag.position())
# print(ag.size())
width, height = ag.size() # 取得螢幕尺寸
print(width, height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
prev_position = None
move_distance = 0
threshold = 9
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark.landmark[9])
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height # 若鏡頭對桌面的話，前面要1-才會上下相反
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
# 拇指指節、拇指指根，嘗試判斷點擊
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
# 中指指尖、指節1、指節2、指根
mid_tip = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
mid_dip = [Hand_Landmark.landmark[11].x, Hand_Landmark.landmark[11].y]
mid_pip = [Hand_Landmark.landmark[10].x, Hand_Landmark.landmark[10].y]
mid_mcp = [Hand_Landmark.landmark[9].x, Hand_Landmark.landmark[9].y]
# 顯示三指尖夾角cos值，試驗用
# print(count_cos(mid_dip, mid_pip, mid_tip))
# 防靜止晃動的移動量判斷
if prev_position is not None:
move_distance = distance_counting(prev_position, (destination_x, destination_y))
if move_distance > threshold:
ag.moveTo(destination_x, destination_y) # 移動鼠標
prev_position = (destination_x, destination_y) # 紀錄現在位置以供下次迴圈判斷移動量
# 進行點擊的判斷
# if click(index_finger, thumb_tip, mid_finger):
# ag.leftClick(destinationX, destinationY)
# 點擊判斷2.0-拇指點擊版本(so far so good)
if click(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destination_x, destination_y)
# 偵測滾輪的捲動
wheel_scroll(mid_dip, mid_tip, mid_pip)
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
cam.release()
cv2.destroyAllWindows
# 檢測中指彎曲
def wheel_determine(mid_point, side_point1, side_point2):
theta = count_cos(mid_point, side_point1, side_point2)
if theta > -0.75: # 判斷為彎曲
return 1
elif theta < -0.9965: # 判斷為伸直
return 2
# 滾輪滾動事件
def wheel_scroll(mid, side1, side2):
if wheel_determine(mid, side1, side2) == 1:
ag.scroll(-10) # 向下捲動
elif wheel_determine(mid, side1, side2) == 2:
ag.scroll(10) # 向上捲動
```
:::
---
## 6/14 Day 10
#### 08:26
廢了三天，老兄要沒料了
目前好像可以弄一下重複點擊的問題，今天下午自主學習可以試試看，還有開始用右鍵好了
幹，還有食指改斜率判斷
接下來在defs中要加入tan的運算和判斷，試試看能不能解決點擊判定不佳的問題
先睡ㄌ
---
## 6/16 Day11
#### 00:45
老兄剛剛想到，<font color='961573'>要用一個設定可以關掉滑鼠功能</font>，也就是在開著程式時可以離開滑鼠而不會打字時一直動到游標
#### 10:33
今天的待處理事項：
1. 小指右鍵
2. 手勢關滑鼠
修改了中指的判定，從指尖、一二指節改成指尖、二指節和指根
### Roger_Try_count_angle(but failed)
:::spoiler
```py=
def finger_angle(v1, v2):
v1_x = v1[0]
v1_y = v1[1]
v2_x = v2[0]
v2_y = v2[1]
try:
angle_ = degrees(acos((v1_x*v2_x+v1_y*v2_y)/(((v1_x**2+v1_y**2)**0.5)*((v2_x**2+v2_y**2)**0.5))))
except:
angle_ = 180
return angle_
def hand_angle(hand_):
angle_list = []
angle_ = finger_angle(
((int(hand_[0][0]) - int(hand_[18][0])), (int(hand_[0][1]) - int(hand_[18][1]))),
((int(hand_[19][0]) - int(hand_[20][0])), (int(hand_[19][1]) - int(hand_[20][1])))
)
def right_click(finger_angle):
small_finger = finger_angle[0]
if small_finger<50:
return 'good'
elif small_finger>=50:
return 'nice'
```
:::
### main
:::spoiler
```py=
```
:::
### definitions
:::spoiler
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from definitions import *
import threading
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三座標算cos用的
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 左鍵點擊判斷的程式
def click_left(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return: whether to click
"""
cos_theta = count_cos(point1, point2, point3)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
# 右鍵點擊
def click_right(p1,p2,p3):
cos_theta = count_cos(p1, p2, p3)
if cos_theta > -0.95: # 這是拇指判斷的值
return 1
# 判斷偵測到的手(左右相反)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 主程式
def hand_detect():
cam = cv2.VideoCapture(0) # 取用相機
cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
# print(ag.position())
# print(ag.size())
width, height = ag.size() # 取得螢幕尺寸
print(width, height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
prev_position = None
move_distance = 0
threshold = 9
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark.landmark[9])
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height # 若鏡頭對桌面的話，前面要1-才會上下相反
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
# 拇指指節、拇指指根，嘗試判斷點擊
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
# 中指指尖、指節1、指節2、指根
mid_tip = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
mid_dip = [Hand_Landmark.landmark[11].x, Hand_Landmark.landmark[11].y]
mid_pip = [Hand_Landmark.landmark[10].x, Hand_Landmark.landmark[10].y]
mid_mcp = [Hand_Landmark.landmark[9].x, Hand_Landmark.landmark[9].y]
pinky_tip =[Hand_Landmark.landmark[20].x, Hand_Landmark.landmark[20].y]
pinky_dip =[Hand_Landmark.landmark[19].x, Hand_Landmark.landmark[19].y]
pinky_pip =[Hand_Landmark.landmark[18].x, Hand_Landmark.landmark[18].y]
pinky_mcp =[Hand_Landmark.landmark[17].x, Hand_Landmark.landmark[17].y]
# 顯示三指尖夾角cos值，試驗用
print(count_cos(pinky_dip, pinky_tip, pinky_pip))
# 防靜止晃動的移動量判斷
if prev_position is not None:
move_distance = distance_counting(prev_position, (destination_x, destination_y))
if move_distance > threshold:
ag.moveTo(destination_x, destination_y) # 移動鼠標
prev_position = (destination_x, destination_y) # 紀錄現在位置以供下次迴圈判斷移動量
# 進行點擊的判斷
# if click(index_finger, thumb_tip, mid_finger):
# ag.leftClick(destinationX, destinationY)
# 點擊判斷2.0-拇指點擊版本(so far so good)
if click_left(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destination_x, destination_y)
if click_right(pinky_dip, pinky_tip, pinky_pip):
ag.rightClick(destination_x, destination_y)
# 偵測滾輪的捲動
#wheel_scroll(mid_pip, mid_tip, mid_mcp)
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
cam.release()
cv2.destroyAllWindows
# 檢測中指彎曲
def wheel_determine(mid_point, side_point1, side_point2):
theta = count_cos(mid_point, side_point1, side_point2)
if theta > -0.76: # 判斷為彎曲
return 1
elif theta < -0.9965: # 判斷為伸直
return 2
# 滾輪滾動事件
def wheel_scroll(mid, side1, side2):
if wheel_determine(mid, side1, side2) == 1:
ag.scroll(-10) # 向下捲動
elif wheel_determine(mid, side1, side2) == 2:
ag.scroll(10) # 向上捲動
```
:::
>勁哥程式貼這邊👇
>[color=#a162cb]
:::spoiler
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from definitions import *
import threading
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三座標算cos用的
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的程式
def click(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return:
"""
cos_theta = count_cos(point1, point2, point3)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
# 判斷偵測到的手(左右相反)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 主程式
def hand_detect():
cam = cv2.VideoCapture(0) # 取用相機
cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
# print(ag.position())
# print(ag.size())
width, height = ag.size() # 取得螢幕尺寸
print(width, height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
prev_position = None
move_distance = 0
threshold = 9
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark.landmark[9])
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height # 若鏡頭對桌面的話，前面要1-才會上下相反
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
# 拇指指節、拇指指根，嘗試判斷點擊
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
# 中指指尖、指節1、指節2、指根
mid_tip = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
mid_dip = [Hand_Landmark.landmark[11].x, Hand_Landmark.landmark[11].y]
mid_pip = [Hand_Landmark.landmark[10].x, Hand_Landmark.landmark[10].y]
mid_mcp = [Hand_Landmark.landmark[9].x, Hand_Landmark.landmark[9].y]
landmarks = Hand_Landmark.landmark
# 顯示三指尖夾角cos值，試驗用
# print(count_cos(mid_dip, mid_pip, mid_tip))
# 防靜止晃動的移動量判斷
if prev_position is not None:
move_distance = distance_counting(prev_position, (destination_x, destination_y))
switcher = 0
if move_distance > threshold:
switcher = gesture_identify(landmarks)
if switcher == 1:
cv2.putText(img, 'close', (30, 120), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255))
switcher_pause = 0
while switcher_pause != 2:
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark.landmark[9])
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height # 若鏡頭對桌面的話，前面要1-才會上下相反
switcher_pause = gesture_identify(Hand_Landmark.landmark)
ag.moveTo(destination_x, destination_y)
cv2.putText(img, 'open', (30, 120), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255))
else:
ag.moveTo(destination_x, destination_y) # 移動鼠標
prev_position = (destination_x, destination_y) # 紀錄現在位置以供下次迴圈判斷移動量
# 進行點擊的判斷
# if click(index_finger, thumb_tip, mid_finger):
# ag.leftClick(destinationX, destinationY)
# 點擊判斷2.0-拇指點擊版本(so far so good)
if click(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destination_x, destination_y)
# 偵測滾輪的捲動
wheel_scroll(mid_dip, mid_tip, mid_pip)
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
cam.release()
cv2.destroyAllWindows
# 檢測中指彎曲
def wheel_determine(mid_point, side_point1, side_point2):
theta = count_cos(mid_point, side_point1, side_point2)
if theta > -0.75: # 判斷為彎曲
return 1
elif theta < -0.9965: # 判斷為伸直
return 2
# 滾輪滾動事件
def wheel_scroll(mid, side1, side2):
if wheel_determine(mid, side1, side2) == 1:
ag.scroll(-10) # 向下捲動
elif wheel_determine(mid, side1, side2) == 2:
ag.scroll(10) # 向上捲動
def gesture_identify(marks):
thumb_mcp = [marks[2].x, marks[2].y]
thumb_ip = [marks[3].x, marks[3].y]
thumb_tip = [marks[4].x, marks[4].y]
ind_pip = [marks[6].x, marks[6].y]
ind_dip = [marks[7].x, marks[7].y]
ind_tip = [marks[8].x, marks[8].y]
mid_pip = [marks[10].x, marks[10].y]
mid_dip = [marks[11].x, marks[11].y]
mid_tip = [marks[12].x, marks[12].y]
ring_pip = [marks[14].x, marks[14].y]
ring_dip = [marks[15].x, marks[15].y]
ring_tip = [marks[16].x, marks[16].y]
pinky_pip = [marks[18].x, marks[18].y]
pinky_dip = [marks[19].x, marks[19].y]
pinky_tip = [marks[20].x, marks[20].y]
thu = wheel_determine(thumb_ip,thumb_tip, thumb_mcp )
ind = wheel_determine(ind_dip, ind_tip, ind_pip)
mid = wheel_determine(mid_dip, mid_tip, mid_pip)
rin = wheel_determine(ring_dip, ring_tip, ring_pip)
pin = wheel_determine(pinky_dip, pinky_tip, pinky_pip)
if thu == 1 and ind == 1 and mid == 2 and rin == 2 and pin == 2:
return 1
elif thu == 1 and ind == 2 and mid == 2 and rin == 2 and pin == 2:
return 2
return 0
```
:::
基本上這個開關弄完就可以demo了
---
## 6/17 Day12
老哥們今天加油，主要完成手勢開關，記得註記一下是什麼手勢，如果還有時間的話，可以把程式直接改成可開可關
關於開關的方式，我這邊有一個想法：
```py=
def gesture_identify(thumb, index, mid, ring, pinky, counter # 用以計算開關的次數):
# 這邊放判斷手勢的程式
if OK: #假如用ＯＫ
counter + = 1
if counter % 2 == 0:
return 1
```
counter負責計算開關次數，若第一次開則會將之變1，並因為是奇數而關閉滑鼠，融入主程式要注意，迴圈要先偵測開關才能進入滑鼠，不然在迴圈內偵測開關會導致關掉之後開不起來
#### 11:17
由於判斷是連續數幀的畫面，根據上面的想法判斷後會發現計數器將重複被記次，導致無法順利開關。改以跳至新迴圈來避免重複觸發的問題，當偵測到手勢，將跳至一個空的無限迴圈，直到無限迴圈再度偵測到手勢才會跳回原程式。
#### 18:50
從勁哥貼的那邊看來，手指彎曲是用wheel_determine檢測，但每隻手指從鏡頭下看來的角度其實不一定一樣，或許每隻手指到彎曲判定再print測試一下後改theta判斷的值就會好一點？（就是每個都獨立有一個角度判斷的事件，因為他們的角度都不一樣）
還有如果要以三點判斷角度的話，可能tip, pip, mcp的夾角會比較明顯。
---
## 6/18 Day13
#### 00:33
明天修手勢的問題，如果手勢修好了但是迴圈上還是不能跑的話，就要引進threading了，threading中有一個功能叫做鎖定(lock)，如果兩個平行執行緒共用同一個鎖定，那麼就只會執行第一個鎖定的執行緒，其他則要等其解鎖後才執行
我們可以利用這個特性，將主程式設為第一個鎖定的執行緒，偵測到手勢後就將其解鎖(lock.release)，則第二執行緒(也就是暫停偵測的部分)執行，在第二執行緒中再度偵測手勢，偵測到後將執行緒一上鎖(lock.acquire)，預計就會再度執行主程式。
<font color='b134ac'>**e04 沒用啊**</font>
早上我再想想，我原本以為上鎖可以這樣用但是好像不行，當解鎖後就只會兩者一起繼續執行，而不是停下第一執行緒
早上再看一下threading.Event，我需要睡眠
輕則思考版本
#### 12:16
其實根本不用那麼麻煩用threading，迴圈可以解決(半夜到底在幹嘛)
概念如下：
```py=
import time
# 原本不論輸入甚麼，都會移動(輸出moving)
# 如果輸入ok(也就是拍到ok)，則停止移動
# 接著輸入其他手勢將會not moving
# 再次輸入ok，也就是再次偵測到ok，則回到移動(moving)
def ok(n):
if n == 'ok':
print('OK')
return 1
else:
return 0
def move():
print('moving')
time.sleep(0.5)
identifier = 0
while True:
gesture = input('輸入手勢：')
if not identifier:
if ok(gesture):
identifier = 1
else:
move()
else:
print('not moving')
if ok(gesture):
identifier = 0
```
這段是以輸入文字代替攝影機拍照，當偵測到ok也就是手勢為ok，一開始輸入任何手勢都會輸出moving，輸入第一次ok後，輸出OK表示有偵測到ok手勢，再次輸入任何手勢則會輸出not moving。
輸入第二次ok後，才會再次回到moving，大概是這個概念。
```mermaid
graph LR;
拍攝手部-- ok-->再次拍攝手部;
拍攝手部-- not ok-->移動;
移動-->拍攝手部;
再次拍攝手部--not ok-->再次拍攝手部;
再次拍攝手部-- ok-->拍攝手部;
```
我這邊改成每次都回到大迴圈，以if, else進行分流，這樣可以確保每次都會拍攝到手部(輸入手勢)，跟勁哥原本的概念不太一樣，上面的程式看起來是偵測到ok後進入一個小迴圈偵測手勢，但小回圈內我沒有看到拍攝的動作，而且再度拍攝就要再寫一次獨入座標的動作，可能程式會有重複，因此如此改動。
### def最新版 6/18
:::spoiler
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from definitions import *
import threading
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三邊長算cos用的
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的主要程式
def click(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return:
"""
cos_theta = count_cos(point1, point2, point3)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
# 判斷偵測到的手(左右相反)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 主程式
def hand_detect():
cam = cv2.VideoCapture(1) # 取用相機
cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
# print(ag.position())
# print(ag.size())
width, height = ag.size() # 取得螢幕尺寸
print(width, height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
prev_position = None
move_distance = 0
threshold = 9
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark)
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height # 若鏡頭對桌面的話，前面要1-才會上下相反
# 食指，中指，大拇指指尖座標的list(點擊判斷用)
index_finger = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
# 拇指指節、拇指指根，嘗試判斷點擊
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
# 食指指尖、指節1、指節2、指根
index_tip = [Hand_Landmark.landmark[8].x, Hand_Landmark.landmark[8].y]
index_dip = [Hand_Landmark.landmark[7].x, Hand_Landmark.landmark[7].y]
index_pip = [Hand_Landmark.landmark[6].x, Hand_Landmark.landmark[6].y]
index_mcp = [Hand_Landmark.landmark[5].x, Hand_Landmark.landmark[5].y]
# 中指指尖、指節1、指節2、指根
mid_tip = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
mid_dip = [Hand_Landmark.landmark[11].x, Hand_Landmark.landmark[11].y]
mid_pip = [Hand_Landmark.landmark[10].x, Hand_Landmark.landmark[10].y]
mid_mcp = [Hand_Landmark.landmark[9].x, Hand_Landmark.landmark[9].y]
# 無名指指尖、指節1、指節2、指根
ring_tip = [Hand_Landmark.landmark[16].x, Hand_Landmark.landmark[16].y]
ring_dip = [Hand_Landmark.landmark[15].x, Hand_Landmark.landmark[15].y]
ring_pip = [Hand_Landmark.landmark[14].x, Hand_Landmark.landmark[14].y]
ring_mcp = [Hand_Landmark.landmark[13].x, Hand_Landmark.landmark[13].y]
# 小指指尖、第一、二指節及指根
pinky_mcp =[Hand_Landmark.landmark[17].x, Hand_Landmark.landmark[17].y]
pinky_pip =[Hand_Landmark.landmark[18].x, Hand_Landmark.landmark[18].y]
pinky_dip =[Hand_Landmark.landmark[19].x, Hand_Landmark.landmark[19].y]
pinky_tip =[Hand_Landmark.landmark[20].x, Hand_Landmark.landmark[20].y]
# 顯示三指尖夾角cos值，試驗用
# print('=============================')
# print(count_cos(thumb_ip, thumb_mcp, thumb_tip))
# print(count_cos(index_pip, index_mcp, mid_tip))
# print(count_cos(mid_pip, mid_mcp, mid_tip))
# print(count_cos(ring_pip, ring_mcp, ring_tip))
# print(count_cos(pinky_pip, pinky_mcp, pinky_tip))
if not identifier:
if gesture_determine(Hand_Landmark.landmark):
identifier = 1
else:
# 防靜止晃動的移動量判斷
if prev_position is not None:
move_distance = distance_counting(prev_position, (destination_x, destination_y))
if move_distance > threshold:
ag.moveTo(destination_x, destination_y) # 移動鼠標
prev_position = (destination_x, destination_y) # 紀錄現在位置以供下次迴圈判斷移動量
# 進行點擊的判斷
# if click(index_finger, thumb_tip, mid_finger):
# ag.leftClick(destinationX, destinationY)
# 點擊判斷2.0-拇指點擊版本(so far so good)
if click(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destination_x, destination_y)
# 偵測滾輪的捲動
wheel_scroll(mid_dip, mid_tip, mid_pip)
else:
if gesture_determine(Hand_Landmark.landmark):
identifier = 0
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
cam.release()
cv2.destroyAllWindows
# 檢測中指彎曲
def wheel_determine(mid_point, side_point1, side_point2):
theta = count_cos(mid_point, side_point1, side_point2)
if theta > -0.75: # 判斷為彎曲
return 1
elif theta < -0.9965: # 判斷為伸直
return 2
# 滾輪滾動事件
def wheel_scroll(mid, side1, side2):
if wheel_determine(mid, side1, side2) == 1:
ag.scroll(-10) # 向下捲動
elif wheel_determine(mid, side1, side2) == 2:
ag.scroll(10) # 向上捲動
'''def count_tangent(point1, point2):
x1, y1, x2, y2 = point1[0], point1[1], point2[0], point2[1]
tangent = '''
def gesture_identify(marks):
thumb_mcp = [marks[2].x, marks[2].y]
thumb_ip = [marks[3].x, marks[3].y]
thumb_tip = [marks[4].x, marks[4].y]
ind_pip = [marks[6].x, marks[6].y]
ind_dip = [marks[7].x, marks[7].y]
ind_tip = [marks[8].x, marks[8].y]
mid_pip = [marks[10].x, marks[10].y]
mid_dip = [marks[11].x, marks[11].y]
mid_tip = [marks[12].x, marks[12].y]
ring_pip = [marks[14].x, marks[14].y]
ring_dip = [marks[15].x, marks[15].y]
ring_tip = [marks[16].x, marks[16].y]
pinky_pip = [marks[18].x, marks[18].y]
pinky_dip = [marks[19].x, marks[19].y]
pinky_tip = [marks[20].x, marks[20].y]
thu = wheel_determine(thumb_ip,thumb_tip, thumb_mcp )
ind = wheel_determine(ind_dip, ind_tip, ind_pip)
mid = wheel_determine(mid_dip, mid_tip, mid_pip)
rin = wheel_determine(ring_dip, ring_tip, ring_pip)
pin = wheel_determine(pinky_dip, pinky_tip, pinky_pip)
if thu == 1 and ind == 1 and mid == 2 and rin == 2 and pin == 2:
return 1
elif thu == 1 and ind == 2 and mid == 2 and rin == 2 and pin == 2:
return 2
return 0
```
:::
等勁哥的手勢判斷
---
## 6/19 Day14
#### 10:37
剩手指角度判斷，昨天試了一下發現ok的角度變化不是很大，在偵測方面要有不靈敏的心理準備
而迴圈的部分已經處理好了，把上次的帶進去了，等總測試確定能用就來demo
然後老兄們先把報告弄出來，程式解說的部分可以先貼圖片就好，不確定拿來幹嘛的部分先留著我之後補
#### 21:30
弄出來了
### main
:::spoiler
```py=
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
from math import *
from definitions import *
import threading
import time
if __name__ == '__main__':
hand_detect()
```
:::
### definitions
:::spoiler
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0]-loc2[0])**2+(loc1[1]-loc2[1])**2)
return distance
# 以三邊長算cos用的
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2)/(2 * a * b)
# 點擊判斷的主要程式
def click(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return:
"""
cos_theta = count_cos(point1, point2, point3)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
# 判斷偵測到的手(左右相反)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 檢測中指彎曲
def wheel_determine(mid_point, side_point1, side_point2):
theta = count_cos(mid_point, side_point1, side_point2)
if theta > -0.75: # 判斷為彎曲
return 1
elif theta < -0.9965: # 判斷為伸直
return 2
# 滾輪滾動事件
def wheel_scroll(mid, side1, side2):
if wheel_determine(mid, side1, side2) == 1:
ag.scroll(-20) # 向下捲動
elif wheel_determine(mid, side1, side2) == 2:
ag.scroll(20) # 向上捲動
# 開關用手勢判斷(OK)
def gesture_identify(marks):
# 拇指節點
thumb_mcp = [marks[2].x, marks[2].y]
thumb_ip = [marks[3].x, marks[3].y]
thumb_tip = [marks[4].x, marks[4].y]
# 食指節點
ind_mcp = [marks[5].x, marks[5].y]
ind_pip = [marks[6].x, marks[6].y]
ind_tip = [marks[8].x, marks[8].y]
# 中指節點
mid_mcp = [marks[9].x, marks[9].y]
mid_pip = [marks[10].x, marks[10].y]
mid_tip = [marks[12].x, marks[12].y]
ring_mcp = [marks[13].x, marks[13].y]
ring_pip = [marks[14].x, marks[14].y]
ring_tip = [marks[16].x, marks[16].y]
# 小指節點
pinky_mcp = [marks[17].x, marks[17].y]
pinky_pip = [marks[18].x, marks[18].y]
pinky_tip = [marks[20].x, marks[20].y]
thu = count_cos(thumb_ip, thumb_tip, thumb_mcp)
ind = count_cos(ind_pip, ind_tip, ind_mcp)
mid = count_cos(mid_pip, mid_tip, mid_mcp)
rin = count_cos(ring_pip, ring_tip, ring_mcp)
pin = count_cos(pinky_pip, pinky_tip, pinky_mcp)
# 拇指、食指：彎曲
# 中指、無名指、小指：伸直
if thu > -0.94 and ind > -0.9 and mid > -0.988 and rin > -0.998 and pin > -0.998:
return 1
else:
return 0
# 主程式
def hand_detect():
cam = cv2.VideoCapture(1) # 取用相機
cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
# print(ag.position())
# print(ag.size())
width, height = ag.size() # 取得螢幕尺寸
print(width, height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
prev_position = None
move_distance = 0
threshold = 9
identifier = 0
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark)
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height
# 拇指指尖、指節、拇指指根，嘗試判斷點擊
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
# 中指指尖、指節1、指節2、指根
mid_tip = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
mid_pip = [Hand_Landmark.landmark[10].x, Hand_Landmark.landmark[10].y]
mid_mcp = [Hand_Landmark.landmark[9].x, Hand_Landmark.landmark[9].y]
if gesture_identify(Hand_Landmark.landmark):
print('ok')
# 手勢開關
if not identifier:
# 可移動時偵測到OK：關閉移動功能
if gesture_identify(Hand_Landmark.landmark):
identifier = 1
print('pause')
time.sleep(1)
# 移動&點擊
else:
# 防靜止晃動的移動量判斷
if prev_position is not None:
move_distance = distance_counting(prev_position, (destination_x, destination_y))
if move_distance > threshold:
ag.moveTo(destination_x, destination_y) # 移動鼠標
# 紀錄現在位置以供下次迴圈判斷移動量
prev_position = (destination_x, destination_y)
# 點擊判斷-拇指點擊版本(so far so good)
if click(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destination_x, destination_y)
# 偵測滾輪的捲動
print(count_cos(mid_pip, mid_tip, mid_mcp))
wheel_scroll(mid_pip, mid_tip, mid_mcp)
else:
if gesture_identify(Hand_Landmark.landmark):
time.sleep(1)
identifier = 0
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
cam.release()
cv2.destroyAllWindows
```
:::
準備demo囉各位
### FINAL main
最後我把definitions跟main合併了，變這樣：
:::spoiler
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0] - loc2[0]) ** 2 + (loc1[1] - loc2[1]) ** 2)
return distance
# 以三邊長算cos用的
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2) / (2 * a * b)
# 左鍵點擊判斷的程式
def click_left(point1, point2, point3):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return: whether to click
"""
cos_theta = count_cos(point1, point2, point3)
if cos_theta > -0.9: # 這是拇指判斷的值
return 1 # 指尖判斷的值是cos_theta<-0.7
# 右鍵點擊
def click_right(p1, p2, p3):
cos_theta = count_cos(p1, p2, p3)
if cos_theta > -0.8: # 這是拇指判斷的值
return 1
# 判斷偵測到的手(左右相反)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 檢測中指彎曲
def wheel_determine(mid_point, side_point1, side_point2):
theta = count_cos(mid_point, side_point1, side_point2)
if theta > -0.75: # 判斷為彎曲
return 1
elif theta < -0.9965: # 判斷為伸直
return 2
# 滾輪滾動事件
def wheel_scroll(mid, side1, side2):
if wheel_determine(mid, side1, side2) == 1:
ag.scroll(-20) # 向下捲動
elif wheel_determine(mid, side1, side2) == 2:
ag.scroll(20) # 向上捲動
# 開關用手勢判斷(OK)
def gesture_identify(marks):
# 拇指節點
thumb_mcp = [marks[2].x, marks[2].y]
thumb_ip = [marks[3].x, marks[3].y]
thumb_tip = [marks[4].x, marks[4].y]
# 食指節點
ind_mcp = [marks[5].x, marks[5].y]
ind_pip = [marks[6].x, marks[6].y]
ind_tip = [marks[8].x, marks[8].y]
# 中指節點
mid_mcp = [marks[9].x, marks[9].y]
mid_pip = [marks[10].x, marks[10].y]
mid_tip = [marks[12].x, marks[12].y]
ring_mcp = [marks[13].x, marks[13].y]
ring_pip = [marks[14].x, marks[14].y]
ring_tip = [marks[16].x, marks[16].y]
# 小指節點
pinky_mcp = [marks[17].x, marks[17].y]
pinky_pip = [marks[18].x, marks[18].y]
pinky_tip = [marks[20].x, marks[20].y]
# 計算手指角度cos值
thu = count_cos(thumb_ip, thumb_tip, thumb_mcp)
ind = count_cos(ind_pip, ind_tip, ind_mcp)
mid = count_cos(mid_pip, mid_tip, mid_mcp)
rin = count_cos(ring_pip, ring_tip, ring_mcp)
pin = count_cos(pinky_pip, pinky_tip, pinky_mcp)
# 拇指、食指：彎曲
# 中指、無名指、小指：伸直
if thu > -0.94 and ind > -0.9 and mid > -0.988 and rin > -0.998 and pin > -0.998:
return 1
else:
return 0
# 主程式
def hand_detect():
cam = cv2.VideoCapture(1) # 取用相機
cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
width, height = ag.size() # 取得螢幕尺寸
print(width, height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
# 為移動閾值準備
prev_position = None
move_distance = 0
threshold = 9
# 為手勢開關準備
identifier = 0
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark)
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height
# 拇指指尖、指節、拇指指根，嘗試判斷點擊
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
# 中指指尖、指節1、指節2、指根
mid_tip = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
mid_pip = [Hand_Landmark.landmark[10].x, Hand_Landmark.landmark[10].y]
mid_mcp = [Hand_Landmark.landmark[9].x, Hand_Landmark.landmark[9].y]
# 小指指尖、指節、指根
pinky_tip = [Hand_Landmark.landmark[20].x, Hand_Landmark.landmark[20].y]
pinky_pip = [Hand_Landmark.landmark[18].x, Hand_Landmark.landmark[18].y]
pinky_mcp = [Hand_Landmark.landmark[17].x, Hand_Landmark.landmark[17].y]
if gesture_identify(Hand_Landmark.landmark):
print('ok')
# 手勢開關
if not identifier:
# 可移動時偵測到OK：關閉移動功能
if gesture_identify(Hand_Landmark.landmark):
identifier = 1
print('pause')
time.sleep(1)
# 移動&點擊
else:
# 防靜止晃動的移動量判斷
if prev_position is not None:
move_distance = distance_counting(prev_position, (destination_x, destination_y))
if move_distance > threshold:
ag.moveTo(destination_x, destination_y) # 移動鼠標
# 紀錄現在位置以供下次迴圈判斷移動量
prev_position = (destination_x, destination_y)
# 點擊判斷-拇指點擊版本(so far so good)
if click_left(thumb_ip, thumb_tip, thumb_mcp):
ag.leftClick(destination_x, destination_y)
if click_right(pinky_pip, pinky_tip, pinky_mcp):
ag.rightClick(destination_x, destination_y)
# 偵測滾輪的捲動
wheel_scroll(mid_pip, mid_tip, mid_mcp)
else:
# 關閉時偵測到OK：開啟移動功能
if gesture_identify(Hand_Landmark.landmark):
time.sleep(1)
identifier = 0
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
cam.release()
cv2.destroyAllWindows
if __name__ == '__main__':
hand_detect()
```
:::
終於可以安穩地睡覺了
*by ELO*
---
## 最終成品
```py=
from math import *
import mediapipe as mp # 手形辨識
import cv2 # 取用相機
import pyautogui as ag # 移動滑鼠
import time
# 很明顯，算距離用的
def distance_counting(loc1, loc2):
distance = sqrt((loc1[0] - loc2[0]) ** 2 + (loc1[1] - loc2[1]) ** 2)
return distance
# 以三邊長算cos用的
def count_cos(mid, side1, side2):
a = distance_counting(mid, side1)
b = distance_counting(mid, side2)
c = distance_counting(side1, side2)
return (a ** 2 + b ** 2 - c ** 2) / (2 * a * b)
# 左鍵點擊判斷的程式
def click_left(point1, point2, point3, position):
"""
:param point1: location of index finger
:param point2: location of thumb
:param point3: location of ring finger
:return: whether to click
"""
cos_theta = count_cos(point1, point2, point3)
# 根據手部位置不同拍攝出角度會不同
if position[0] > 0.5: # 相機右側
if cos_theta > -0.85: # 這是拇指判斷的值
return 1
else: # 相機左側
if cos_theta > -0.9: # 這是拇指判斷的值
return 1
# 右鍵點擊
def click_right(p1, p2, p3, position):
cos_theta = count_cos(p1, p2, p3)
# 根據手部位置不同拍攝出角度會不同
if position[0] > 0.5: # 相機右側
if cos_theta > -0.79: # 這是小拇指判斷的值
return 1
else: # 相機左側
if cos_theta > -0.7: # 這是小拇指判斷的值
return 1
# 判斷偵測到的手(左右相反)
def side_detect(handedness):
for idx, hand_handedness in enumerate(handedness):
return hand_handedness.classification[0].label
# 檢測中指彎曲
def wheel_determine(mid_point, side_point1, side_point2):
theta = count_cos(mid_point, side_point1, side_point2)
if theta > -0.75: # 判斷為彎曲
return 1
elif theta < -0.9965: # 判斷為伸直
return 2
# 滾輪滾動事件
def wheel_scroll(mid, side1, side2):
if wheel_determine(mid, side1, side2) == 1:
ag.scroll(-20) # 向下捲動
elif wheel_determine(mid, side1, side2) == 2:
ag.scroll(20) # 向上捲動
# 開關用手勢判斷(OK)
def gesture_identify(marks):
# 拇指節點
thumb_mcp = [marks[2].x, marks[2].y]
thumb_ip = [marks[3].x, marks[3].y]
thumb_tip = [marks[4].x, marks[4].y]
# 食指節點
ind_mcp = [marks[5].x, marks[5].y]
ind_pip = [marks[6].x, marks[6].y]
ind_tip = [marks[8].x, marks[8].y]
# 中指節點
mid_mcp = [marks[9].x, marks[9].y]
mid_pip = [marks[10].x, marks[10].y]
mid_tip = [marks[12].x, marks[12].y]
ring_mcp = [marks[13].x, marks[13].y]
ring_pip = [marks[14].x, marks[14].y]
ring_tip = [marks[16].x, marks[16].y]
# 小指節點
pinky_mcp = [marks[17].x, marks[17].y]
pinky_pip = [marks[18].x, marks[18].y]
pinky_tip = [marks[20].x, marks[20].y]
# 計算手指角度cos值
thu = count_cos(thumb_ip, thumb_tip, thumb_mcp)
ind = count_cos(ind_pip, ind_tip, ind_mcp)
mid = count_cos(mid_pip, mid_tip, mid_mcp)
rin = count_cos(ring_pip, ring_tip, ring_mcp)
pin = count_cos(pinky_pip, pinky_tip, pinky_mcp)
# 拇指、食指：彎曲
# 中指、無名指、小指：伸直
if thu > -0.94 and ind > -0.9 and mid > -0.988 and rin > -0.998 and pin > -0.998:
return 1
else:
return 0
# 主程式
def hand_detect():
cam = cv2.VideoCapture(1) # 取用相機
cam.set(cv2.CAP_PROP_FPS, 120)
ag.PAUSE = 0.001
width, height = ag.size() # 取得螢幕尺寸
print(width, height)
mp_hands = mp.solutions.hands # 手掌偵測方法
mp_drawing = mp.solutions.drawing_utils # 繪圖方法
mp_drawing_styles = mp.solutions.drawing_styles # 繪圖樣式
ag.FAILSAFE = False
# 為移動閾值準備
prev_position = None
move_distance = 0
threshold = 9
# 為手勢開關準備
identifier = 0
# 啟用手掌偵測
with mp_hands.Hands(
model_complexity=1,
min_tracking_confidence=0.5,
min_detection_confidence=0.5) as hands:
while True:
rec, img = cam.read()
if not rec: # 如果打不開鏡頭
print('BRUH')
break
img = cv2.resize(img, (800, 450))
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # CV2讀入的顏色tuple是BGR，要轉換成RGB
results = hands.process(img2) # 識別手部
# 辨識右手(滑鼠限右手用)(左撇子不友善:((
if results.multi_handedness:
# 將手部辨識的結果(左右手)讀入
if side_detect(results.multi_handedness) == 'Left': # 這邊寫left是因為我的鏡頭左右翻轉
if results.multi_hand_landmarks:
# 將手部辨識結果(座標)讀入，並轉成螢幕的座標
for ind, Hand_Landmark in enumerate(results.multi_hand_landmarks):
# print(Hand_Landmark)
destination_x = ((6 / 4) - (Hand_Landmark.landmark[9].x / 0.4)) * width # 9是中指指根
destination_y = ((6 / 4) - (Hand_Landmark.landmark[9].y / 0.4)) * height
# 拇指指尖、指節、拇指指根，嘗試判斷點擊
thumb_tip = [Hand_Landmark.landmark[4].x, Hand_Landmark.landmark[4].y]
thumb_ip = [Hand_Landmark.landmark[3].x, Hand_Landmark.landmark[3].y]
thumb_mcp = [Hand_Landmark.landmark[2].x, Hand_Landmark.landmark[2].y]
# 中指指尖、指節1、指節2、指根
mid_tip = [Hand_Landmark.landmark[12].x, Hand_Landmark.landmark[12].y]
mid_pip = [Hand_Landmark.landmark[10].x, Hand_Landmark.landmark[10].y]
mid_mcp = [Hand_Landmark.landmark[9].x, Hand_Landmark.landmark[9].y]
# 小指指尖、指節、指根
pinky_tip = [Hand_Landmark.landmark[20].x, Hand_Landmark.landmark[20].y]
pinky_pip = [Hand_Landmark.landmark[18].x, Hand_Landmark.landmark[18].y]
pinky_mcp = [Hand_Landmark.landmark[17].x, Hand_Landmark.landmark[17].y]
if gesture_identify(Hand_Landmark.landmark):
print('ok')
# 手勢開關
if not identifier:
# 可移動時偵測到OK：關閉移動功能
if gesture_identify(Hand_Landmark.landmark):
identifier = 1
print('pause')
time.sleep(1)
# 移動&點擊
else:
# 防靜止晃動的移動量判斷
if prev_position is not None:
move_distance = distance_counting(prev_position, (destination_x, destination_y))
if move_distance > threshold:
ag.moveTo(destination_x, destination_y) # 移動鼠標
# 紀錄現在位置以供下次迴圈判斷移動量
prev_position = (destination_x, destination_y)
# 點擊判斷-拇指點擊版本(so far so good)
if click_left(thumb_ip, thumb_tip, thumb_mcp, (destination_x, destination_y)):
ag.leftClick(destination_x, destination_y)
time.sleep(0.05)
print(count_cos(thumb_ip, thumb_tip, thumb_mcp))
if click_right(pinky_pip, pinky_tip, pinky_mcp, (destination_x, destination_y)):
ag.rightClick(destination_x, destination_y)
time.sleep(0.05)
# 偵測滾輪的捲動
wheel_scroll(mid_pip, mid_tip, mid_mcp)
else:
# 關閉時偵測到OK：開啟移動功能
if gesture_identify(Hand_Landmark.landmark):
time.sleep(1)
identifier = 0
# 將手部識別點繪製於拍攝畫面上
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(img,
hand_landmarks,
mp_hands.HAND_CONNECTIONS,
mp_drawing_styles.get_default_hand_landmarks_style(),
mp_drawing_styles.get_default_hand_connections_style())
cv2.imshow('ROGER BASILISK V4 GAMING MOUSE', img) # 顯示拍攝畫面
if cv2.waitKey(5) == ord('q'):
break
cam.release()
cv2.destroyAllWindows
if __name__ == '__main__':
hand_detect()
```
---
## 待解問題
1. - [ ] 精準點擊
2. - [x] 較好的點擊
3. - [ ] 更好的移動依據(慢來，反正能動就先別改
4. - [x] 移動涵蓋螢幕
5. - [x] 在移動到邊緣時手指仍在鏡頭內
6. - [ ] 鼠標穩定性提升
7. - [ ] 食指操縱點擊
8. - [x] 滾輪功能
9. - [x] 右鍵功能
10. - [x] <font color="5ec653">滑鼠的手勢開關</font>
11. - [x] <font color="fa4cdaw">never gonna give you up. never gonna let you down</font>
12. - [ ] 新環境和角度下的校正功能
## 參考網站
可用參考網站：
1. mediapipe & autogui合用指南 http://www.ctimes.com.tw/DispArt-tw.asp?O=HK75TA82N5CARASTD3
2. 幾乎跟我們要做的東西一樣的中國網站 https://blog.csdn.net/Already8888/article/details/126125467
3. 手部辨識的座標原理 https://blog.csdn.net/weixin_45930948/article/details/115444916
4. 手勢快捷，貌似挺有料 https://medium.com/jimmy-wang/mediapipe-hands-%E6%93%8D%E4%BD%9C%E8%AA%AA%E6%98%8E-%E5%AF%A6%E6%88%B0%E7%AF%87-%E4%B8%8B-e273bda92c48
5. pyautogui 中文教學 https://ithelp.ithome.com.tw/articles/10277668?sc=iThelpR #老哥，這玩意兒可以拿來按popcat :wth
6. mediapipe 手部辨識教學 https://ithelp.ithome.com.tw/m/articles/10299964
7. class教學 https://steam.oxxostudio.tw/category/python/basic/class.html
8. thread教學(平行化執行緒) https://steam.oxxostudio.tw/category/python/library/threading.html
9. hackmd教學 https://hackmd.io/@Kawaii-kanataso/Hackmd_tutorial?fbclid=lwAR3AWX-UWrnqkaBk5PddtFhXl2OFoSkhe0ZuwYfS0Ue00i7lp0gjq_aLglw_aem_th_AZIDMgTMo92k9HBiDPVZOuF1aY8CBgymUL2MelpYhhDFe8T8KwU_1w_abGRwR5T9lwU&mibextid=S66gvF
Sign in with Wallet
Connect another wallet