***Automatic Optical Inspection And Robotic Arm Control Based On Deep Learning***
===
## **Intro**
* Industrial automation technology is the trend development of today's society, such a rapidly changing technology gradually flooded our life, the production side expects to produce a larger number of products at a lower cost, but also expects that through automation technology, greatly reduce the use of human resources, improve the overall production efficiency and reduce production costs.
* In this paper, we use YOLO (You Only Look Once), a deep learning technology for object detection, as a cornerstone to combine a self-made 3D printing robotic arm with a self-built neural network to implement automatic control of the robotic arm and automatic optical detection and to roughly understand the details of automation technology.
* The End-to-End algorithm can avoid the disadvantage of traditional object detection that must be trained separately, and significantly accelerate the computation speed and reduce our computation time in the automation process.
* The end-to-end algorithm can avoid the disadvantage of traditional object detection that must be trained separately, and significantly speed up the computation speed and reduce our computation time in automation. In addition, we also build our own neural network to predict the motor angle of each axis of the six-axis robotic arm based on the 256 positions of data we collected as input points, which can be used as our software to control the robotic arm.
## **Abstract**
* This project is designed to automate the production line of a hardware parts factory, and to develop and integrate software and hardware technologies with a special focus on the operation of general screw production lines.
* In this project, three sizes of screws and nuts are used as production components. We will develop the technology of positioning and identification of screws and nuts based on deep learning technology, so that the computer can monitor the positioning of screws and nuts of different sizes and different angles and directions placed on the production line workbench through a camera, and can distinguish which size of screws or nuts they are.
* After the positioning and differentiation is done, the robot arm is controlled to precisely clamp the specified size of screw or nut from the worktable. For this purpose, this project will use a 3D printer to make and assemble the robot arm components, and use Arduino with control circuit design to make a six-axis robot arm that can be operated in practice.
---
## **Processes**
* Step 1 : **Take a Picture**
Use the webcam to capture platform images and save them to your computer.
* Step 2 : **YOLO**
Put the saved images into the YOLO model and let the model grab the features and distinguish the screws or nuts.
* Step 3 : **Recognize Objects**
Use Yolo to obtain the identified location coordinates and the length and width of the object bouning box.
* Step 4 : **Classifier**
The binary decision tree method is used to create the classifier.
* Step 5 : **Recognize the Size of Objects**
Use the classifier to identify the size of objects, YOLO is unable to identify the size of objects.
* Step 6 : **Predict Moter Angle**
Using the neural network K-Nearest Neighbors (KNN), the motor angles of the six axes are predicted based on the object coordinates.
* Step 7 : **Catch Objects**
The parameters derived from the classifier and the neural network are passed to the Arduino, which controls the clamping of objects and their classification by size.
* Step 8 : **Repeat**
Go back to Step 1 until no more objects exist on the screen.。
---
## **Graph of Processes**
---
## Systems of Development
### Hardware
* Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
* GEFORCE GTX 1080 Ti
* Arduino Uno
* 3D Printed Robotic Arm
### Software
* Ubuntu 20.04
* Python Language
* C/C++ Language
* Opencv
* Scikit - Learn
---
## **Techniques**
### **YOLO**
YOLO is an object recognition algorithm implemented by CNN, using CNN to predict multiple bounding-boxes at the same time and calculate the probability of the object for each box, and in training is also directly to take the whole picture and throw it into the CNN for training, so end-to-end algorithm can avoid the disadvantage of traditional object detection that must be trained separately, and significantly accelerate the speed of computing.
YOLO Reference : https://github.com/AlexeyAB
CNN Reference : https://reurl.cc/qmyZDp
#### LabelImg
YOLO is a type of supervised learning in the field of machine learning, so all training data must be labeled.
Reference : https://github.com/tzutalin/labelImg
#### Training Results
---
### **Binary Decision Tree**
We put the length and width of the BoundingBox of the object identified by YOLO into this Binary Decision Tree to identify the size of the object.
* **Requirement**
* **csv**
* **Pandas**
* **Sklearn**
* **Numpy**
Python File : https://github.com/Aaron-Ace/DLAOIBOT/blob/main/decisionTree.py
Binary Decision Tree Reference : https://medium.com/@Packt_Pub/binary-decision-trees-1ec94cfed208
---
### **Arduino**
Arduino is a set of code that can be programmed in the Arduino compiler environment to the Arduino development board, which will then send electronic signals to the servo motor to achieve the effect we want, and thus drive the robot arm.
Arduino Reference : https://blog.jmaker.com.tw/arduino-tutorials-1/
* **Requirement** (The Arduino environment is built in Anaconda, so the environment is installed using conda install.)
Anaconda Official Website : https://www.anaconda.com/products/individual
* **Python**
* **Pyserial**
```
conda create -n arduino python
```
```
conda install pyserial
```
* **Pycharm**
The data transfer between Arduino's compiler environment and Pycharm needs to be mediated by Pyserial, and the Python compiler environment needs to be preceded by a header file.
Pycharm Official Website : https://www.jetbrains.com/pycharm/
```python
import serial
```
Python File : https://github.com/Aaron-Ace/DLAOIBOT/blob/main/armcontrol.py
* **Arduino IDE**
Remember to add this header file in order to transfer data with Pycharm.
```c
#include <SoftwareSerial.h>
```
This header file is required for the servo motor.
```c
#include <Servo.h>
```
---
### **Attend Regressor**
We use this neural network model to transform the object position (x, y) derived from the YOLO model into the coordinates (u, v) on the platform by using Perspective Transform. This (u, v) is put into the neural network, and the 6 axis angles of the robot arm are derived.
* **Requirement**
* **Python**
* **Numpy**
* **Opencv**
* **Pandas**
* **Tensorflow**
* **Perspective Transform**
Perspective Transform can re-calibrate any four points in the picture into a rectangle by using them as reference points.
Python File : https://github.com/Aaron-Ace/DLAOIBOT/blob/main/XY2UV.py
```python
src_points = np.array([[25, 39], [711, 38], [27, 715], [706, 714]],np.float32) #(x, y)
dst_points = np.array([[-20, -20], [-20, 20], [20, -20], [20, 20]],np.float32) #(x, y)
```
The src_points are the reference points (you can adjust them according to your needs), and calibrate them to dst_points as the desired coordinates.
```python
M = cv2.getPerspectiveTransform(src_points, dst_points)
```
Using this function, the resulting M is the conversion matrix.
Perspective Transform Reference : https://www.pyimagesearch.com/2014/08/25/4-point-opencv-getperspective-transform-example/
* **Attend Regressor Model**
Python File : https://github.com/Aaron-Ace/DLAOIBOT/tree/main/MLP
About 250 pieces of data were collected, including about 200 pieces of training data and about 50 pieces of measurement data.
---
### **Robotic Arm**
The arms of the machine are made by ourselves, and the 3D printer is used to print out the required parts first, and then assembled.
Layout : https://github.com/Aaron-Ace/DLAOIBOT/tree/main/RoboticArm3DModel
3D Printer : https://www.3dprow.com/products/snapmaker-20-a250
Arduino Code : https://github.com/Aaron-Ace/DLAOIBOT/blob/main/ArduinoCode/ArduinoCode.ino
---
## **Usage**
### Command
* Clone the Git
```
git clone https://github.com/Aaron-Ace/DLAOIBOT.git
```
* Run Main Program
```
python main.py
```
### GUI
* Run GUI
```
python GUI.py
```
* The two Check buttons on the top are camera and robot arm connection confirmation, once the connection is successful, 'Sucessful' will appear.
* The middle five options are to choose whether to clip the object and its size.
* AutoCorrect : Automatic correction between the screen and the platform to avoid accidental shaking of the camera
Code：https://github.com/Aaron-Ace/DLAOIBOT/blob/main/Correction.py
* ForceStop： Compulsory Termination
* Execute : Run
---
## **Conclusion**
* **Accurancy**
* **YOLO Accurancy** : Screw 99.6% Nut 99.5%
* **Catching Accurancy** : Screw 95.4% Nut 92.9%
* **Classification Accurancy** : Screw 97.5% Nut 95.0%
* **Video** : https://youtu.be/YGjaYCNhIYg
* **Platfrom**
Sign in with Wallet
Connect another wallet