--- title: Info2021 Homework 4 tags: 資訊科技產業專案設計, interview --- # ISP/NPU Algorithm Team >[老師回饋](https://youtu.be/YVbXyjWS71o?t=7913) >ISP 的準備方向可參考 Github 3A 開源專案 >並可看[並行和多執行緒程式設計](https://hackmd.io/@sysprog/concurrency/https%3A%2F%2Fhackmd.io%2F%40sysprog%2FS1AMIFt0D) >NPU 競爭者少可研究相關議題來爭取機會 ## What is Image Signal Processor [ISP (影像處理) 簡述及調適應用](http://www.synnex.com.cn/component/tech_paper_12.html) >ISP是Image Signal Processor的縮寫,ISP影像信號處理器的主要作用是對前端sensor (影像傳感器) 輸出的信號做後期處理。不同的ISP用來搭配不同廠商的sensor。ISP在整個 camera產品中很重要,它直接影響呈現給用戶的畫質優劣。圖像經過CCD或者CMOS的採集後,需要經過後期處理才可以較好的適應不同的環境,在不同光學條件下都能較好的還原出人眼所見的真實細節。在ISP中它會完成我們常常提及的3A(AWB/AE/AF),自動白平衡/自動曝光/自動對焦)。傳統模式下一般採用一顆DSP或者一顆FPGA來完成對圖像的後期處理。有些相機產品支援3D降噪功能、寬動態、慢快門、幀累積、強光抑制等功能也都是ISP進行處理。 [ISP(影象訊號處理)演算法概述、工作原理、架構、處理流程](https://www.gushiciku.cn/pl/p7wM/zh-tw) > ISP是Image Signal Processor 的簡稱,也就是影象訊號處理器。 > >ISP一般用來處理Image Sensor(影象感測器)的輸出資料,如做AEC(自動曝光控制)、AGC(自動增益控制)、AWB(自動白平衡)、色彩校正、Lens Shading、Gamma 校正、祛除壞點、Auto Black Level、Auto White Level等等功能的處理。 > >DSP是Digital Signal Processor 的縮寫,也就是數字訊號處理器。 >>而DSP功能就比較多了,它可以做些拍照以及回顯(JPEG的編解碼)、錄影以及回放(Video 的編解碼)、H.264的編解碼、還有很多其他方面的處理,總之是處理數字訊號了。ISP是一類特殊的處理影象訊號的DSP。 [翻新ISP設計?-徐宏民](https://www.digitimes.com.tw/col/article.asp?id=1199) > 近幾年興起的卷積網路(CNN)能否取代傳統ISP的工作?能否設計一個CNN將影像感測訊號(如RAW原始格式)直接轉換為高品質的全彩影像?這是近年學術界新興的問題...... ## AI on chip [硬核科普!Modem、ISP、DSP、NPU等單元到底有啥用?](https://read01.com/2KkjDeA.html) >DSP的跨界應用 >>隨著VR/AR應用,以及AI人工智慧的興起,高通賦予了DSP單元更多能力,其中就包括向量擴展(HVX)和Tensor張量加速器。 沒辦法,驍龍SoC一直都沒有集成獨立的NPU單元,其主打的「高通人工智慧引擎AI Engine」需要CPU、GPU、DSP、內存、緩存等模塊協同作戰。而其他晶片商的SoC很早以前就引入了NPU模塊,所以無需讓DSP進行太大的改變。 > >NPU——聰明靠它 >>2017年底,華為推出的麒麟970第一次引入了NPU(神經網絡處理器)概念,讓SoC也具備了更強的本地(端側)AI運算能力(類似於「硬解」),執行效率可以秒殺CPU的「軟解」。至此,NPU單元就與AI劃上了等號。 > >無處不在的AI >>最開始,AI主要還是用於拍照,隨後,AI功能逐漸拓展,幾乎所有的應用場景都離不開AI加速運算。 [AI芯片,APU/BPU/CPU/DPU/FPU/HPU/GPU/TPU/NPU/MPU/MCU/VPU/的区别](https://www.daimajiaoliu.com/daima/485da8f681003fe) [在這個很多人為「NPU」而興奮的當下,我這有盆冷水想潑](https://kknews.cc/tech/22rvp69.html) [深度學習軟硬體加速器探索](https://ictjournal.itri.org.tw/content/Messagess/contents.aspx?PView=1&MmmID=654304432061644411&MSID=1001517114142321327) ## 相關工作職缺項目 ### [Google ISP job](https://careers.google.com/jobs/results/?distance=50&location=Taiwan&q=ISP) 台灣是 Google Silicon Team 重要據點,Tensor 晶片開發的職缺很多。其中徵了許多 ML/CV 的職缺,應是 ISP/TPU 的實作研發。 > [Google's Tensor inside of Pixel 6, Pixel 6 Pro: A Look into Performance & Efficiency](https://www.anandtech.com/show/17032/tensor-soc-performance-efficiency) > Pixel Visual Core / Neural Core / Tensor ... :::spoiler #### [Software Engineer, Machine Learning, Pixel Camera](https://careers.google.com/jobs/results/122490584670577350-software-engineer-machine-learning-pixel-camera/?distance=50&location=Taiwan&q=ISP) > [Software Engineer, Computer Vision, Pixel Camera](https://careers.google.com/jobs/results/122520332381627078-software-engineer-computer-vision-pixel-camera/?distance=50&location=Taiwan&q=ISP) **Minimum qualifications:** - Bachelor's degree in Computer Science or related technical field or equivalent practical experience. - 2 years of work or educational experience in Machine Learning or Artificial Intelligence. - 1 year of relevant experience, including software development. - Experience with one or more general purpose programming languages, including but not limited to: Java, C/C++ or Python **Preferred qualifications:** - PhD or equivalent practical experience in Electrical Engineering, Computer Science, or related field. - Extensive practical experience with Machine Learning in the context of image processing. - Experience with Software/Hardware ISP pipeline optimization. - Experience with building infrastructure and tools for adjusting and rating images (web front-end development). **Responsibilities** - Develop the core image and video processing algorithms to enable better photography and videography experiences on Android platform. - Work with the system team in order to ship great products. #### [Image Signal Processor System Modeling Engineer, Silicon](https://careers.google.com/jobs/results/92132500513399494-image-signal-processor-system-modeling-engineer-silicon/?distance=50&location=Taiwan&q=ISP) **Minimum qualifications:** - Bachelor's degree in Computer Science, Electrical Engineering, or equivalent practical experience - 2 years of industry experience or equivalent. **Preferred qualifications:** - Master's degree in Computer Science or Electrical Engineering. - Experience in SoC development environment. - Experience in reference model development. - Solid knowledge of image, video and computer vision processing algorithms. - Proficiency in C or C++ or Python programming. **Responsibilities** - Utilize software engineering and programming skills in C/C++, Python. - Create and maintain software architecture for next generation ISP system. Implement, model, analyze and test ISP algorithm pipeline. - Develop software environment for ISP system development and simulation. - Support delivery of ISP c-models for implementation, verification and SoC system. Integrate the ISP c-models into tuning and pre-silicon software environment. - Collaborate in a team environment to build system level designs and methods. #### [Image Quality Engineer, Chrome OS](https://careers.google.com/jobs/results/134346399002043078-image-quality-engineer-chrome-os/?distance=50&location=Taiwan&q=ISP) **Minimum qualifications:** - Bachelor's degree in Electrical Engineering, Computer Science, Mathematics, Physics, or equivalent practical experience. - Experience in image quality tuning for consumer cameras. **Preferred qualifications:** - Master's degree or PhD in Electrical Engineering, Computer Science, Mathematics, Physics, or equivalent. - 8 years of imaging tuning related experience. - Experience with auto focus, auto exposure, and auto white balance algorithms. - Experience with auto focus, auto exposure and auto white balance algorithms. - Experience with ISP pipeline algorithms. - Understanding of imaging system, from sensor/lens to final images. **Responsibilities** - Manage and validate Chrome OS projects and make sure the image quality (IQ) is at a reasonable level. - Work with internal team to define image quality strategy, challenge and target to achieve team missions. - Work with partners to identify the needs in order to achieve their image quality target. - Provide documents, guides, and consultations to partners and check test data and reports in order to qualify the camera. ::: ### [Qualcomm ISP job](https://qualcomm.dejobs.org/jobs/?q=ISP&location=Taiwan) Qualcomm ISP 在台灣都是徵 Senior to Staff ,ISP 應不是台灣 Qualcomm 主要的項目,在此穿插一則南韓的 Junior 職缺。 :::spoiler #### [Camera ISP Algorithm System Engineer, Senior to Staff in Taipei, Taiwan](https://qualcomm.dejobs.org/taipei-twn/camera-isp-algorithm-system-engineer-senior-to-staff/894684DD4CF14294839C03B9CAD4835A/job/) **Responsibilities** - Research and development for next generation camera features and image quality improvement - Prototype solutions on mobile platforms for proof of concept and demonstration purposes - Deployment of solutions that improve the mobile imaging experience. - Design of camera systems that balance power, memory, silicon area to maximize imaging performance **Basic Qualifications** - 5+ years working experiences in the following areas: - Signal/image processing and IQ tuning experience - Camera and image processing technologies - CMOS sensors and algorithms for sensor and optics processing - Android HAL system design - Multi-camera system design including at least one of following subjects or similar: temporal synchronization, image fusion, depth estimation, dual-camera calibration. - Knowledge of modern machine learning techniques - Customer support/ camera tuning experience - C, C++, Matlab, Python **Preferred Qualifications** - Candidates should have excellent analytical and problem solving skills along with strong coding background - Track record of driving ideas from design through commercialization - Excellent knowledge of C++, capability to design and implement robust, high-performance and flexible system software - Experience in integrating machine learning algorithm into camera systems. - 10% oversea travel is required #### [Camera ISP Algorithm Designer - Camera (QCT) in Suwon, South Korea](https://qualcomm.dejobs.org/suwon-kor/camera-isp-algorithm-designer-camera-qct/179BF5C6A54B43D89FC4099750F07926/job/) **Responsibilities** - Research and development for mobile camera features and image/video quality improvement - Prototype solutions on platforms for proof of concept and demonstration purposes - Deployment of solutions that improve the mobile imaging experience. - Design of camera systems that balance power, memory, silicon area to maximize imaging performance. **Minimum Qualifications** 3+ years industry experiences in the following areas: - Experience with signal, image and color processing - Experience with C, C++, Matlab - Excellent analytical and problem-solving skills **Preferred Qualifications** - Over Master Degree (PHD Degree preferred) - Experience with camera technology and ISP (Image Signal Processor) pipeline design - Experience with different types of sensors and algorithms for sensor and optics processing - Experience with mobile imaging related development and technology - Driving ideas from design through commercialization **Keywords** > ISP, Image Processing, CMOS Sensor, Algorithm, Color Processing, Noise Reduction, Detail Enhancement, Local Tone Mapping, HDR ::: ### [Mediatek ISP job](https://careers.mediatek.com/eREC/JobSearch?searchKey=ISP#) > [聯發科5G旗艦晶片天璣9000亮相!首採台積電4奈米、應用手機明年Q1登場](https://www.bnext.com.tw/article/66275/mediatek-dimensity-9000-5g-soc) MM1, MM2 多媒體部門 Camera 影像處理。 :::spoiler #### [Camera image processing and 3A algorithm engineer](https://careers.mediatek.com/eREC/JobSearch/JobDetail/MTK120160303000?langKey=en-US) **Job Description** - Camera 3A Algorithm Development - Camera 3A Control Flow / HAL Development - Camera 3A Tuning and Customer Support **Requirement** - MS degree in Optical, Electrical Engineering, Computer Science, Imaging or related field or equivalent practical experience. - Knowledge of image sensor, image quality, tuning, calibration, camera modules - Familiarity with C/C++ - Experience developing Android middleware/Camera HAL or ISP/3A algorithm design or tuning is a plus #### [Camera IQ Tuning Engineer 2021](https://careers.mediatek.com/eREC/JobSearch/JobDetail/MKR120210630000?langKey=en-US) **Job Description** - Responsible for supporting customer on Android Camera System - Responsible for image tuning based on Android Camera System, Image Sensor, 3A, ISP - Responsible for supporting customer about technical discussion, issue review, analysis and debugging image quality issue - Co-work and discuss with HQ engineer and customer to support project and issue **Requirement** - BS/MS in Computer Science, Computer Engineering or Electrical Engineering, Image Processing - Minimum 2 years of experience in camera ISP/3A tuning development - Well knowledge of camera tuning (3A, EE/NR/Gamma/CCM) - Well knowledge of Camera Technology, Image Processing - Familiar with C & C++ language. - Good verbal and written communication skills in English #### [Machine Learning and Computer Vision Algorithm Engineer (AI)](https://careers.mediatek.com/eREC/JobSearch/JobDetail/MTK120160906004?langKey=en-US) > 偷渡一個和 AI/CV 相關的 offer. **Job Description** 1. AI Vision Algorithm Development 2. AI Video Algorithm Development 3. Edge platform-aware Algorithm Development and Optimization 4. Advanced AI Algorithm Development (1) Neural network optimization (2) Neural network design automation (3) Self-supervised/unsupervised learning (4) Continuous learning 5. Edge AI SW platform development **Requirement** 1. Familiar with deep learning framework: Tensorflow, Pytorch, Caffe/Caffe2. 2. Familiar with on-device framework for mobile/embedding devices: TFLite, ANN. 3. Required programming language: Python, C/C++, Java. 4. 2+ years experience in deep learning algorithm/model development/optimization, model training methodology. 5. Have experience in developing deep learning algorithms for mobile/embedding devices. 6. Familiar with software development and software engineering, version control system #### [Real-time Video Processing/AI Algorithm](https://careers.mediatek.com/eREC/JobSearch/JobDetail/MTK120200821008?langKey=en-US) **Job Description** We are the algorithm team to develop video processing algorithms for improving the picture quality gap from content to display. The algorithm topics are in the fields of image processing, computer vision, and deep learning. - Develop image and video processing algorithm. - Develop firmware of picture quality algorithm. - Co-work with hardware designer to realize algorithm on chip. - Co-work with customers to develop customized picture quality algorithm. **Requirement** - Must be familiar with image/video processing algorithm - Must be familiar with C or C++ programming skills. - Better to have knowledge of deep learning. - Better to have knowledge of computer architecture, digital hardware design, embedded system. #### [Multimedia Algorithm Development Engineer](https://careers.mediatek.com/eREC/JobSearch/JobDetail/MTK120191127001?langKey=en-US) **Job Description** - Develop real-time video processing algorithm for display of TV/monitor - Develop firmware of picture quality algorithm for TV/monitor SoC. - Co-work with hardware designer to realize algorithm on chip. - Co-work with customers to develop customized picture quality algorithm. **Requirement** - Be familiar with image/video processing. - Strong C/C++ programming skills. - Better to have knowledge of computer architecture, digital hardware design, embedded system. - Better to have knowledge of Linux/Unix OS. #### [Multimedia Software Engineer](https://careers.mediatek.com/eREC/JobSearch/JobDetail/MTK120160822002?langKey=en-US) > [徵才: 聯發科相機軟體開發工程師](https://www.dcard.tw/f/tech_job/p/236741348) **Job Description** - Camera/ISP Driver Development - Camera Framework Development - Camera Tuning Flow Development **Requirement** - Strong C/C++ programming and debug skills - Nice to have ARM linux embedded system programming experience - Nice to have Camera Development experience - Nice to have IC Verification experience ::: ### [Novatek Algorithm Job](https://www.104.com.tw/company/12nopku0?roleJobCat=1_2007001012&area=0&page=1&pageSize=20&order=8&asc=0&tab=job) 聯詠也在徵求 AI/ML 的職缺,當時我實習的任務就有點像 **NN/DL影像視訊處理研發工程師** 的工作內容。實習時主管希望我能補充計算機結構/數位 IC 設計的背景知識,在此份職缺的JD 也提到**具備DLA/NPU 相關設計經驗者尤佳**。 - [RISC-V Virtual Platform-Based Convolutional Neural Network Accelerator Implemented in SystemC](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiT75OBm6v0AhUOr1YBHSgoAswQFnoECAQQAQ&url=https%3A%2F%2Fwww.mdpi.com%2F2079-9292%2F10%2F13%2F1514%2Fpdf&usg=AOvVaw1JhVdY8-mUaeoUWVeI8gWu) :::spoiler #### [影像演算法工程師(A2)](https://www.104.com.tw/job/5bi32?jobsource=company_job) **【產品線描述】** 大尺寸 Mini./Micro LED 拼接屏Tcon 開發。 **【工作說明】** 1. 數位相機擷取/影像處理/演算法開發 2. 具3A Algorithm/Image Pipeline/或其它演算法開發 3. LED拼接屏系統架構開發 4. 客戶端之on site design in工程支援。 **【必要條件】** 碩士以上電子﹑ 電機、光電相關系所畢業。 **其他條件** 熟悉或修習過下列任何一種技術者優先考慮: 1. 熟影像處理相關演算法設計者。 2. 具OLED面板或驅動IC設計/驗證相關經驗者。 3. 熟悉半導體元件的模型分析與量測 。 4. 具MCU相關開發經驗或架設自動量測儀器開發經驗者。 #### [NN/DL影像視訊處理研發工程師](https://www.104.com.tw/job/66uof?jobsource=company_job) **【產品線描述】** Smart TV Solutions:提供TVSoC、MEMC/FRC及面板相關顯示裝置的控制晶片 ASIC Solutions:提供智能手機、智能電視、電競螢幕及商用顯示等產品各種ASIC解決方案 PD and Gaming Solutions:提供各種商用顯示及電競螢幕的控制晶片 **【工作說明】** 本職缺內容主要為研究與開發視訊處理與電腦視覺相關的演算法或硬體架構。 利用深度學習或其他先進的演算法,改良現有的視訊處理效果與開發更先進的影像視訊處理方法, 並且結合演算法與硬體知識,進一步最佳化深度學習的效能。 1. 利用Neural network來處理影像視訊的優化 2. 收集dataset來訓練網路 3. 最佳化model來符合硬體架構實現 4. DLA的架構規劃與設計 5. DLA相關的開發工具, 網路驗證, 與最佳化工具 **【必要條件】** 1. MS/Ph.D. in CS/EE (or related major) 2. 熟悉 深度學習的概念, 演算法 與架構 3. 具深度學習工具的使用經驗. (ex. python, tensorflow, pytorch, caffe,....) **其他條件** 1. 具備DLA/NPU 相關設計經驗者尤佳 2. 熟悉 C/C++ 程式設計者尤佳 ::: ## Assessment ### ISP/NPU Job - 外商大多需要有 ISP 相關工作經驗者(可從一線 IC Algorithm and Architecture team 尋找機會) - 了解 ISP Pipeline 與 數位影像處理的基礎。 - Android or Linux Embedded System 實作經驗 - Computer Architecture (NPU 更強調這塊的 knowhow) - ~~Digital Hardware Design(有點底層)~~ - 一些 ML knowledge。 ### Self-assessment - 純血四大電資 - 具備 CV / ML 相關背景,修習「數位影像」,曾撰寫 Visual C++ 專案實作 edge detection, image registration, Connected Component Cluster, Histogram Equalization 等演算法。 - 曾在聯詠演算法部門實習經驗(ML for video artifact reduction) - 缺乏 ISP 背景知識與系統開發經驗 - 缺乏嵌入式系統開發經驗 - 缺乏 Computer Architecture 背景 ## 模擬面試 🧔:interviewer 👶:interviewee ### 背景詢問 🧔:詢問關於數位影像實作的內容 🧔:詢問關於聯詠演算法實習的內容 ### 背景知識 #### DIP [How Blurs & Filters Work - Computerphile](https://www.youtube.com/watch?v=C_zFhWdM4ic) 🧔:如何消除噪點。 👶:mean / median filter。大多使用 Gaussian Filter, 🧔:如何強化邊緣? 👶:請問 Laplacian of Gaussian(二階微分), Sobel Filter(一階微分), Canny Edge Detection 。 #### DSP 🧔:請你解釋一下傅立葉分析。 [圖解傅立葉分析](https://hackmd.io/@sysprog/fourier-transform) #### ISP 🧔:ISP 的Pipeline。 🧔:什麼是 3A > [camera 知識 - 3A](http://jay75328.blogspot.com/2015/05/camera-3a.html) 👶:AF(Auto Focus, 自動對焦), AE(Auto Exposure, 自動曝光), AWB (Auto White Balance自動白平衡) **AWB :** 自動白平衡的功用是對因光源色溫變化而造成色差的現象做校正,例如黃昏時的戶外,因色溫較低,所以感應器感應到的顏色會偏黃,而白天的色溫較高,所以感應器感應到的顏色會偏藍(請參考以下色溫與其對應色偏顏色的示意圖) **AE :** 自動曝光則是用在拍攝環境的亮度不同時,以控制光圈或快門的方式,達到適當曝光的目的,使得影像不致過亮或過暗,但是考慮成本的問題,一般PC Cam的鏡頭為固定光圈,並且省略機械快門,而由感應器以電子快門的方式來控制曝光。 **AF :** 自動對焦的處理,對手動對焦的鏡頭而言可被省略,但是對自動對焦的鏡頭就相當重要了,因為它關係到影像的清晰度。 🧔:什麼是 Codec 👶:[以前做的 codec 筆記](https://www.notion.so/artifact-reduction-f8c2688c1a1a4a96a76a727d6e65165a) #### OS 🧔:Process 和 Thread 的差別 🧔:解釋 DeadLock 🧔:semaphore和MutexLock #### Computer Architecture 🧔:解釋 Hazard 👶:Structual Hazard, Data Hazard, Control Hazard 🧔:介紹 Pipeline ## 相關面試整理 ### C/C++ - [C/C++ - 常見 C 語言觀念題目總整理(適合考試和面試)](https://mropengate.blogspot.com/2017/08/cc-c.html) - [C面試考題](https://hackmd.io/@a110605/By6DscbVM?type=view) - [【面試】C語言面試100題](https://www.twblogs.net/a/5d1fb2e3bd9eee1ede063648) ### OS / Computer Architecture - [面試整理](https://hackmd.io/@g9tdU4gDSTiEZrerd0g7-w/SyCXEfsSE?type=view#%E9%9D%A2%E8%A9%A6%E6%95%B4%E7%90%86) - [作業系統OS](https://hackmd.io/@Pl-eQT9CQaS0jhExKqL8_w/BkhOSR4jW?type=view#%E4%BD%9C%E6%A5%AD%E7%B3%BB%E7%B5%B1OS) - [計算機組織結構](https://hackmd.io/@8bFA57f7SRG-K7AAT8s62g/ryv1NT3S) :::info [[面試] 研替面試心得 聯發科/聯詠/瑞昱](https://www.ptt.cc/bbs/Tech_Job/M.1472189407.A.993.html) 聯發科 職缺:多媒體演算法、軟韌體工程師 1. 上機考C,選擇五題、填充五題、寫code五題(應該沒記錯),時間50分鐘。完全寫不完,平常要有在保持手感才寫得快。 2. 沒有英文成績,所以要做英文測驗,聽力30題、閱讀選擇30題,一樣50分鐘,聽寫都不難。 3. 當天一共有三場面試,一次都是3,4個部門面談,所以小弟就不照職缺細分了。流程:投影片→不時穿插主管的提問→主管追加問題→部門介紹→我問問題約1.5hr。主要的問題都會圍繞著投影片的內容,研究的方法、演算法的細節、用了甚麼工具等。 4. 問題(因為跟投影片內容很相關,僅供參考) [DSP/通訊] * convolution coding / channel coding(完全不會) * Huffman coding * 為甚麼訊號要載高頻傳送? [其他] * 網路系統架構有哪七層 * socket封包在哪一層傳送 * 做過最大的軟硬體project,code 約幾行 ::: :::info [私立學四大碩面試心得](https://www.dcard.tw/f/tech_job/p/234160233) 10.聯發科 camera ISP/ sensor driver 一面 當天報到完後,先去電腦室考C/C++ coding 考題是 sort 以及class相關 考完後被帶到面談室等待面談主管 等了好一陣子 主管來了之後請我自我介紹 之後便介紹該部門 主要是做照相機的ISP 有聊到原先MTK有自己的spec之類的 但最近想follow 某一standard,所以要改寫以前的東西 最近滿花心力滿累的,希望忙完這些事後明年能輕鬆一些 也問了我修課相關 以及我做過的專題 細問了os修課,作業做了哪些 同步相關問題 mutex 跟 semaphore的細節與差異 有binary semaphore 幹嘛還需要mutex等等 我也問了主管很多問題,主要圍繞在上面提及的大工程 之後就結束了面談 :::