owned this note   changed 6 years ago  
Linked with GitHub

Mobile app for sign language translation and learning by image recognition - Claire Chung

由於場地問題，第二天我們移動到另一棟大樓啦！議程教室變動請見網站上的議程表。

歡迎來到 https://hackmd.io/@coscup/2019 共筆

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

點擊本頁上方的 開始用 Markdown 一起寫筆記！
手機版請點選上方 按鈕展開議程列表。

請從這裡開始

Why Sign Translation Mobile App?

  • Sign Translation Mobile App
    • Handy
    • Facilitate communication
    • Real-time real-person interaction
  • Building of the App
    • Data collection
      • Maching Learning、
      • Browse, filter & label signs for PoC
      • Example video of HKSL(Hong Kong Sign Language) signs…
        • Coke, Go Online…etc
    • Model training
    • Interface Design…

Major data sources

  • HKSL Browser
  • LSD Visual

Deep Learning on signs

  • CNN for Image Recognition、RNN for Video streaming

  • About Resource for each approach

    • Tensorflow + FireBase
      • e.q: AutoML Vision Edge
    • Swift with Core ML by Apple for iOS
    • MACE (Mobile AI Compute Engine)
      • By XiaoMi (小米)
    • Flutter

  • Consideration

    • pretrained model

  • Example: Train image model with AutoML Vision Edge in ML Kit

Challenge

  • Lack of ground truth data

  • Accuracy

  • Ambiguity from multi-model signs with overlapped mode in the broad daily use vocabulary

    • Large dataset & much data curation needed
    • Apply RNN & SL version of Word2Vec?

  • CALL FOR HELP

    • Data & tran mdel to be more comprehensive
    • develop better backend model
    • Additional features:
      • e.q: Combine AR for learning
tags: COSCUP2019 跨境合作 - HKOSCon (Open Source HK) IB302
Select a repo