# Research Journal: Thesis Study Report 💓
###### tags: `school notes` `journal`

### Interval time 06062022 - 06302022

### Objectives Done
- [x] Kinova communication on UBUNTU
- [x] Ubuntu C++
- [x] Kinova Movement using C++
- [x] Kinova speed reduction
- [x] Obstacles Simulation
- [x] Save data to CSV
- [x] Trajectory planning block
- [x] RRT simulation with the same path start to goal
- [x] SSM approach
- [x] Velocity decomposition block
- [x] Setting the workspace area
- [x] Homogeneous transform
- [x] Presentation Explanation for SSM innovation
- [x] Camera connection
- [x] Data distance collection
- [x] Data human psychological
- [x] C++ opencv install
- [x] Connect driver openni2 opencv orbbec Astra
- [x] Introduce Proposed approach: Mathematical approach, Block Diagram, Pseudocode, Report
- [x] SSM research Milestones

## 06062022 KINOVA UBUNTU
### Keyword: kinova, ubuntu, communication
### Objectives
- [x] Kinova communication on UBUNTU
- [x] Ubuntu C++
- [x] Kinova Movement using C++
- [x] Kinova speed reduction
- [x] Obstacles Simulation
- [x] Save data to CSV
- [x] Trajectory planning block
### Notes
1. Compile C++ in ubuntu
* Compile ubuntu with library

    g++ -o setpos_kinova -I Kinova setpos_kinova.cpp -lm Kinova.cpp -ldl

* Experiment compiler code

    Compiler for set start position
    g++ -o setpos_kinova -I Kinova setpos_kinova.cpp -lm Kinova.cpp -ldl

    Compiler for Algoritm test
    g++ -o SSM_build -I Kinova SSM_build.cpp -lm Kinova.cpp -ldl
    
    Compiler for no obstacles
    g++ -o SSM_build_noobst -I Kinova SSM_build_noobst.cpp -lm Kinova.cpp -ldl

2. Kinova specification movement
* Translation velocity: 0.04 m/s - 0.2 m/s
* Orientation velocity: 0.1 Rad/s - 1.07 Rad/s
4. Research scenario
* Kinova home position:
Update Robot Position → x:0.211 y:-0.268 z:0.503
Update Robot Orientation → Rx:1.671 Ry:1.105 Rz:0.113
* Start point
Robot Position → x:0.314 y:-0.439 z:0.157
Update Robot Orientation → Rx:2.915 Ry:0.148 Rz:-0.990
* Goal point
Newest Robot Position → x:-0.080 y:-0.439 z:0.157
Newest Robot Orientation → Rx:2.915 Ry:0.148 Rz:-0.990
5. Trajectory Planning Program
**RRT Class**
1. Vareable independent MAX_iteration, max_distance, goal
2. Declare the minimum obstacles obstXmin, obstYmin, obstZmin
3. Declare the maximum obstacles obstXmax, obstYmax, obstZmax
4. Struct Node posXYZ, node *prev, *next
5. Class RRT
1. Private
- Node start, goal, rrtNodes
2. Public
- RRT input startX, startY, startZ, goalX, goalY, goalZ → initialize
- Check obstacles x1,y1,z1, z1, y2, y3 → decision
- Check nearestNode → new_node
- LookForPath
-
3. RRT simulation can work :heavy_check_mark:
### Problem
- [ ] RRT simulation with the same path start to goal
- [ ] SSM approach
- [ ] Velocity decomposition block
### Summary
ALHAMDULILLAH stay progress ya mas bro. Perdalam theoritical approach untuk menceritakan thesis lebih detail

---

## 06072022 Experiment Scenario
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [x] Velocity decomposition block
- [x] Setting the workspace area

### Notes
1. Experiment Worskpace Description
* distance based command 0.394 m
* distance based real calculation 0.40 m
* The error distance 0.01 m

a. First experiment Scenario movement along X-axis
* The robot movement without obstacles
* The robot movement with static obstacles
* The robot movement with dynamic obstacles

b. Second experiment scenario movement along Z-axis
* Input the human physiological -> head, upper body, lower body
* The robot can move with limited speed based on HRC regulations

c. Homogeneous Transform Calculation
* Homogeneous trasnform is used for calculation the robot frame to world frame
* The workspace is rectangle with size (x,y,z) → (0.5327, 0.334, 0.4918)
* The workspace based on robot space
* (minX, maxX) → (-0.1300, 0.4027)
* (minY, maxY) → (-0.2193, -0.5533)
* (minZ, maxZ) → (0.0927, 0.5845)

2. Timing case study
a. Measure execution time based on c++
* include library chrono
* using namespace std → chrono
* Reference → [Time execution](https://www.geeksforgeeks.org/measure-execution-time-function-cpp/)

3. SSM Calculation and Velocity map
*
4. Logical approach for coding Control System Algorithm
1. Prepare for the input data
2. Looping condition for the case
3. Check the robot position
4. Update the input data
5. Control algorithm -> depend on the research case
6. Send command to robot control
6. Logical approach for new device communication
1. Determine the research objectives
2. Learn the spesifications
3. Learn do doesn't
4. Install driver
5. Choose the programming language that can used for developing the task
6. Learn from examples
8. Experiment for subjective safety based on physiological apporach
* Experiment for safety velocity limitation
1. Head
2. Upper body
3.
10.

### Problem
- [x] Homogeneous transform
- [x] Presentation Explanation for SSM innovation
- [ ] RRT simulation with the same path start to goal
- [x] SSM approach

### Summary
- Revise the second paragraph from the motivation. The paragraph explain more about the psychological impact because of HRC and why we need safety on HRC.
- Review again the weakness of SSM and its innovation
- Why we need the trajectory planning
- Limitation depend on the experiment that you will explain in discussion

---

## 06082022 SSM code experiment
### Keywords: SSM and code

### Objectives
- [x] SSM approach

### Notes
1. SSM Definition
References [SSM basic implementation](https://nvlpubs.nist.gov/nistpubs/ir/2012/NIST.IR.7851.pdf)
2. SSM mathematical approach
Equation (1) $$S = K_{H}(T_{R}+T_{B})+K_{R}(T_{R})+B+C$$
Where:
S = Minimum separation distance equation
K_{H} = Speed of human
K_{R} = Speed of robot
T_{R} = Reaction time to detect human and issue a stop – a parameter measured during timing test
T_{B} = Brake time – see below
B = Brake distance – see below
C = C_{H}+C_{R}, the region surrounding the human and robot respectively. For the testbed, this region includes the uncertainty in position and dimension of each.

For SSM testbed, the brake distance is:
$B = \frac{K^2_R}{2A}$
where A is a worst-case deceleration level measured during stopping test, and the brake time is:
$$T_{B}=K_{R}/A$$
The SSM issues a stop if the distance minus the minimum separation
distance, S, goes negative Figure 5.
The human maximum fixed speed 2.0 m/s.

**The important information to obtain the robot reaction time**

**Sensor Parameter**

**Method SSM**

3. SSM Validation
Validation SSM based on the distance experiment
1. Variable we need to experiment (Y-axis) based on time (X-axis)
a. Robot Velocity (m/s)
b. Distance (m)
c. Minimum separation distance
d. contact threshold
2.

4. Review thesis from the title Real-time Human Robot Collaboration and Speed Control of a Robotic Arm for COllaborative Operations [Reference](https://www.researchgate.net/publication/326717734_Real_Time_Human_Robot_Interactions_and_Speed_Control_of_a_Robotic_Arm_for_Collaborative_Operations)
**a. Abstract**
+ cobots increase flexibility, productivity, and safety of the production line
+ Issues safety guarantee is the biggest issue when deploying these systems in industrial environment
+ External vision help to create a real-time adaptive environment. The technical approach robot, sensor, and the algorithm is used.
+ The result and experiment how to demonstate the safety on workspace

**b. Introduction**
* Background
* Background
* trend industrial change to HRC
* the advantages of HRC than traditional industrial robotic
* new robotic regulation for safety issues
* limitation of the interactive tasks
* Goals
* consider own goal
* Initial Work
* the story explain about lab condition and robot UR10 used for the researh
* sensor input kinect v2 camera
* application matlab
* Thesis outline
* consider the own outline

**c. Literature Review**
+ The 3D Camera: Kinect
+ Explanation regarding the kinect spesification
+ Distance calculation form 3D camera
+ Coversion to Cartesian space
+ Communication
+ Skeletal Tracking for human detection
+ Robotic Arm: UR10
+ Robot specification
+ Communication
+ Speed control -> robot performance change the speed time interval
+ Safety and stop
+ Kinematic Model
+ Foward kinematic
+ Invers kinematic
+ Distance Calculation
+ Homogenous transform
+ Translation
+ Rotation
+ Transformation
+ Distance calculation
+ Human Robot Interaction
+ The introduction of HRI
+ Types of interaction
+ ISO Standard
+ Economic motivation
+ Stopping Functions
+ Collaborative modes

**d. Experiment SSM**
+ Setup environment
+ UR10 simulator
+ procedure movement and comparison real and simulation
+ Code structure
+ Safety Rated monited stop
+ Speed and separation monitoring (SSM)
+

**e. Conclusion**
+ Finished
+ Future research need attention

### Problem
- [x] Homogeneous transform
- [x] Presentation Explanation for SSM innovation
- [ ] RRT simulation with the same path start to goal

### Summary

### References
[ISO/TS 15066:2016](http://106.38.59.21:8080/userfiles/d894aa13635f4f70ae241466d6504b6e/files/teckSolution/2019/12/ISO-TS-15066-2016-Robots-and-robotic-devices-%E2%80%94-Collaborative-robots.pdf
)

## 06092022 Depth Camera Explore
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [ ] Camera connection
- [ ] Data distance collection
- [ ] Data human psychological
- [x] Introduce Proposed approach: Mathematical approach, Block Diagram, Pseudocode, Report

### Notes
1. Install driver ubuntu obbrec camera
a. Download driver in this link [driver choose AstraSDK](https://orbbec3d.com/index/download.html) -> linux
b. Extract to the specific folder
c. Install astra from folder install -> open terminal -> sudo sh ./install.sh
d. Install the dependencies -> sudo apt-get install libsfml-dev
e. make the execute file using -> sudo chmod 777 *
f. Run the code using the execute program

2. Install C++ openCV -> [Reff](http://www.codebind.com/cpp-tutorial/install-opencv-ubuntu-cpp/) :heavy_check_mark:
a. Compile c++ opencv library

    g++ main.cpp -o output `pkg-config --cflags --libs opencv`

3. Sample code for check the opencv camera c++ [References](https://www.selfmadetechie.com/how-to-create-a-webcam-video-capture-using-opencv-c)
4. Install libudev.so.1 ->sudo apt install --reinstall libudev1
5. Pseudocode tips trick to introduce the proposed approach
* The main constructs of pseudocode
* SEQUENCE
* WHILE
* REPEAT-UNTIL
* FOR
* IF-THEN-ELSE
* CASE
* Example:
* The other case two most command needed
* Invoking classes or calling functions (using the CALL keyword).
* Handling exceptions (using EXCEPTION, WHEN keywords)
* Example: 
* Rules of the writing pseudocode
1. Always **capitalize** the initial word (often one of the main 6 constructs).
2. Have only **one statement** per line.
3. **Indent** to show hierarchy, improve readability, and show nested constructs.
4. Always **end** multiline sections using any of the END keywords (ENDIF, ENDWHILE, etc.).
5. Keep your statements programming language **independent**.
6. Use the naming domain of the **problem**, not that of the **implementation**. E.g., “Append the last name to the first name” instead of “name = first+ last.”
7. Keep it **simple**, **concise**, and **readable**.
* Example: 
* Final Thoughts
* Writing pseucode is important for working on complex environment.
* It can help us to understand the code and realize the actual code
* It can save more time and effort on fixing bugs and avoiding errors
* It can also communicate more efficiently with others from different backgrounds, as it delivers the algorithm's idea without the complexity of syntax restrictions.

### Problem

### Summary

### References
* I found the book for learning basic programming. Here is the [1](https://github.com/SuperCV/Book)
[2](https://www.mdpi.com/1424-8220/21/21/7144)
* Safety classification [3](https://link.springer.com/content/pdf/10.1007/978-3-030-69178-3.pdf)
* Pseudocode introduction [4](https://towardsdatascience.com/pseudocode-101-an-introduction-to-writing-good-pseudocode-1331cb855be7)

## 06102022 Integrated System in the Workspace
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [x] SSM research Milestones
- [x] Homogeneous transform
### Notes
1. The notes summary in the [thesis report](https://docs.google.com/spreadsheets/d/1mYknPUi8l43HdiqNTZMg_u_7kWAwD3e_c0GFUks-Kp0/edit?usp=sharing)

2. Homogenous transform C++ environment
* we need to install eigen for easier calculation [this site](https://eigen.tuxfamily.org/dox/GettingStarted.html)
* install using command "sudo apt install libeigen3-dev"
* check the eigen library in ubuntu location default "/usr/include/eigen3"
* if the library include, we can compile without mention the file location -> g++ latihan1.cpp -o latihan1
* trigonometry in code using the radian. We need to convert from degree to radia [references](https://cplusplus.com/reference/cmath/sin/)
3. Homogeneous theoritical transform
* There are three kind of the step that we need to consider
1. The transformation form
2. Rotation form
3. Convert to degree

### Problem

### Summary

## 06112022 Integrated System in the Workspace
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [x] Research proposed simulation
### Notes
1. Install coppeliasim on ubuntu
* download file on site [coppeliasim Edu](https://www.coppeliarobotics.com/downloads)
* add the root of vref on /.bashrc -> export VREP_ROOT=/home/wen/Manually_Installed/VREF
* open the sh file using -> ./coppeliaSim.sh
* the coppelia success for opening

2. Translation and rotation coppelia
* need to understanding how to move to parent frame, world frame
3.
### Problem

### Summary

## 06112022 Integrated System in the Workspace
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [x] Presentation approach
### Notes
1. Propose approach presentation preparation
* material preparation for presentation
* introduction
* contribution
* milestones innovation
* propose approach
* experiment scenario
* result estimation
* proposed approach detail
* overview system
* block system explaination
* algorithm
*
### Problem
1. Evaluation from presentation
2. Writing thesis again
### Summary

## 06132022 Evaluation for Meeting Presentation
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [x] presentation point
- [ ] algorithm in pseudocode
- [x] literature review report
### Notes
1. The presentation evaluation on proposed research
* The proposed step
* Review the original paper that you wanna be adopted
* Understand the block system which build the system
* Q1: why we need the block
* Q2: what is the weakness of the system
* Q3: which block are we needed to improve
* You can proposed research with this question for consideration
* Q1: why we need the block
* Q2: what is the input and output that we can use
* Q3: is it cooperate with the original block
* The proposed presentation
* Q1: Perancangan safety harus disesuaikan dengan rules ISO. Apakah ISO memperbolehkan untuk robot tetap berjalan pada saat manusia dekat dengan robot
* Q2: Seperti apa data masukan pada sistem dan data yang ingin dikeluarkan. Bagaimana proses matematis untuk mendapatkan data tersebut

### Problem

### Summary

## 06142022 Integrated System in the Workspace
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [ ] Chapter 2 and 3 outline
- [ ] xx

### Notes
1. Update outline for chapter 2 and 3
2. Try to empasize the research propose
3. Focus on how to get the data and how to elaborate with the original block
### Problem

### Summary

## 06152022 Elaborate The Pattern to Get The System Idea
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [ ] mathematical approach each diagram block
- [ ]
### Notes
1. Figure out the weaknees of current system and evaluate the performance
* SSM method stop when the operator reached the closest area
* Robot-human work with

3. Find out the system literature
4. Mathematical approach for reasoning the propose approach
### Problem

### Summary

## 06172022 - 06202022 Understanding ISO/TS 15066
### Keywords: ISO, safety regulations

### Objectives
- [ ] ISO standards
- [ ] The collaboration operation weakness
- [ ]
### Notes
1. Q1: What is the regulations in HRC?
2. Q2: What is the collaboration operation? How is it work?
3. Q3: What is the weakness for each operation
4. Q4:

**Human Skeleton Logic Implementation**
1. Berdasarkan paper BlazePose. Jelaskan topologi dari keypoint. Output dari keypoint tersebut adalah 33 titik pada manusia
2. Extract data on Joint [video tutorial](https://www.youtube.com/watch?v=06TE_U21FK4) -> output for each landmark is x,y,z, visibility
3.

### Problem
- Bagaimana merancang safety framework HRC pada saat ruangkerja kolaborasi memperbolehkan robot tetap bekerja namun tetap dapat meningkatkan produktivitas
-
### Summary
- [Safety Regulations BSI](http://106.38.59.21:8080/userfiles/d894aa13635f4f70ae241466d6504b6e/files/teckSolution/2019/12/ISO-TS-15066-2016-Robots-and-robotic-devices-%E2%80%94-Collaborative-robots.pdf)
- [Robot Collaboration](https://drive.google.com/file/d/1LVdIhdx0azatWzneFjMnsUJf_dxlHWrh/view)
- [Human Skeleton](https://arxiv.org/pdf/2006.10204.pdf)
-

## 06212022 - 06222022 Analyze mediapipe data Skeleton
### Keywords: Skeleton input SSM

### Objectives
- [x] Install mediapipe
- [x] Run mediapipe skeleton on webcam
- [x] Data characteristic
- [ ] Mathematic approach

### Notes
1. Install mediapipe on ubuntu -> reference: [mediapipe step](https://github.com/feitgemel/Jetson-Nano-Python/blob/master/Install-MediaPipe/How%20to%20Install%20MediaPipe%20on%20jetson-nano%202022.txt)
2. Sudah beres di linux. Catatan menyusul sekarang fokus ke pembuatan rumus setiap tahapan

**Tahapan Riset Mathematical approach**
1. Pembacaan skeleton $q_{hn} = \left \{ n, x_n, y_n, z_n, vis_n \right \}$
dimana $q_{hn}$ merupakan data skeleton ke n
* $n$: skeleton id
* $x_n$ : x position from joint skeleton-n
* $y_n$ : y position from joint skeleton-n
* $z_n$ : z position from joint skeleton-n
* $vis_n$ : visibility from joint skeleon-n

2. Skeleton Extraction
Data skeleton diambil pada beberapa bagian untuk diproses.
* $q_{h0}$ yang merupakan keypoint nose
* $q_{h11}$ dan $q_{h12}$ yang merupakan keypoint right_shoulder dan left_shoulder
* $q_{h23}$ dan $q_{h24}$ yang merupakan keypoint right_hip dan left_hip
* Kelima data tersebut digunakan sebagai data masukan untuk mendapatkan lokasi wajah dan chest dengan rumus euclidean distance berikut $d = \sqrt{(x_1-x_2)^2+(y_1-y_2)^2+(z_1-z_2)^2}$
* Serta untuk mendapatkan nilai tengah skeleton tersebut menggunakan rumus $(x_M, y_M, z_M) = (\frac{x_1+x_2}{2} , \frac{y_1+y_2}{2} , \frac{z_1+z_2}{2})$
* Terdapat tinggi area face dan chest
* PR: memastikan nilai deteksi z dan lokasi berdasarkan realtime
* Lihat errornya jarak kamera deteksi dengan kenyataan: percobaan 1
* membuat skenario simulasi dan ditampilkan ke layar deteksi
* Deteksi skeleton ✔️
* Lokasi face dan chest ✔️
* Perhitungan human velocity ✔️
* Perhitungan SSM dan simulasi area masuk ✔️
* Pengaruh jarak terhadap kecepatan
* Percobaan robot pada area vurnerable
3. Homogeneous transform untuk mendapatkan deteksi
4. Penentuan jarak z sebagai simulasi robot
5. Record data csv and check the graph

### Problem

### Summary

## 06232022 SSM Evaluation and Cooking Idea
### Keywords: SSM improve performance

### Objectives
- [x] SSM current research
- [x] Skeleton Innovation
- [x] How to integrated and maximize the separation distance area
- [ ]
### Notes
1. Arah cerita problem
* Perkembangan teknologi yang memungkinan manusia bekerjasama dengan robot
* Update regulasi terkait safety HRC
* SSM memungkinkan bekerjasama namun pada level collaboration masih kurang efektif karena robot berhenti
* SSM dapat digunakan dengan mengkombinasi PFL
* PFL berfungsi apabila ada tumbukan robot berhenti oleh karena itu membatasi velocity pada area tertentu
* Banyak inovasi yang kurang fleksibel seperti penggunaan marker pada saat kolaborasi
* Pembatasan kecepatan pada area kerja dapat memungkinkan HRC bekerjasama dalam jarak dekat
2. Propose approach
* SSM calculation
* Human detection
* Human skeleton extraction
* Human distance calculation
*
### Problem

### Summary

## 06242022 - 06302022 Integrated System in the Workspace
### Keywords: SSM, RRT approach method, presentation

### Objectives
- [x] Idea for Vmax
- [x] safety analyze the weakness of SSM
- [ ] Presentation improvement SSM on collaboration level

### Notes
1. Coppelia simulation for human robot collaboration scenario
2. Overview System approach

  graph TD;
      A[Human Detection]-->B[Human skeleton selection];
      C[Robot position]-->D[Robot working area];
      B-->E[Distance Calculation];
      E-->G[SSM Calculation];
      D-->E;
      E-->F[Vulnerable Velocity Limitation];
      F-->J;
      G-->H[Free Collision];
      G-->I[Velocity Reduction];
      G-->J[Collaboration Area];
      J-->K[Post-Collision Strategy];

3. Mathematical approach
* Human detection
$Ph_n = \left \{ n, x_n, y_n, z_n, vis_n \right \}$
* dimana $Ph_n$ merupakan data skeleton ke n
* $n$ adalah id dari skeleton
* $x_n$ adalah posisi x dari skeleton ke n
* $y_n$ adalah posisi y dari skeleton ke n
* $z_n$ adalah posisi z dari skeleton ke n
* $vis_n$ adalah presentase terbaca (visibility) skeleton
* Human skeleton selection
Data skeleton diambil pada beberapa bagian untuk diproses.
1. Pembacaan skeleton $q_{hn} = \left \{ n, x_n, y_n, z_n, vis_n \right \}$
dimana $q_{hn}$ merupakan data skeleton ke n
* $n$ adalah id dari skeleton
* $x_n$ adalah posisi x dari skeleton ke n
* $y_n$ adalah posisi y dari skeleton ke n
* $z_n$ adalah posisi z dari skeleton ke n
* $vis_n$ adalah presentase terbaca (visibility) skeleton

2. Skeleton Extraction
Data skeleton diambil pada beberapa bagian untuk diproses.
* $P_{h0}$ yang merupakan keypoint nose
* $P_{h11}$ dan $q_{h12}$ yang merupakan keypoint right_shoulder dan left_shoulder
* $P_{h23}$ dan $q_{h24}$ yang merupakan keypoint right_hip dan left_hip
* Kelima data tersebut digunakan sebagai data masukan untuk mendapatkan lokasi wajah dan chest dengan rumus euclidean distance berikut $d = \sqrt{(x_1-x_2)^2+(y_1-y_2)^2+(z_1-z_2)^2}$
* Serta untuk mendapatkan nilai tengah skeleton tersebut menggunakan rumus $(x_M, y_M, z_M) = (\frac{x_1+x_2}{2} , \frac{y_1+y_2}{2} , \frac{z_1+z_2}{2})$
* Terdapat tinggi area face dan chest
* PR: memastikan nilai deteksi z dan lokasi berdasarkan realtime
* Lihat errornya jarak kamera deteksi dengan kenyataan: percobaan 1
* membuat skenario simulasi dan ditampilkan ke layar deteksi
* Robot position -> $P_{rn} = \left \{ x_n, y_n, z_n, Rx_n, Ry_n, Rz_n \right \}$
* Robot working area -> distance end effector from base and make a circle area
* Area distance calculation
* Velocity limitation based on physical characteristics
* SSM formula based on ISO $S_p(t_0)=S_h+S_r+S_s+C+Z_d+Z_r$
* Second SSM formula ->
* $S_h=\int_{\tau=t_0}^{\tau=t_0+T_R+T_S}V_h(\tau)d_\tau$
* $S_r=\int_{\tau=t_0}^{\tau=t_0+T_R}V_r(\tau)d_\tau$
* $S_s=\int_{\tau=t_0+T_R}^{\tau=t_0+T_R+T_S}V_s(\tau)d_\tau$
* $S_p(t_0)=V_h(T_r+T_s)+V_rT_r+S_s+C+Z$
* $S_s=V_r(t_0)T_s+a_{max}\frac{T_s^2}{2}$
* $T_s= \frac{V_r}{a_{max}}$
* $V_{rmax}(t_0) = \frac{S_p(t_0)-V_h(t_0)(T_r+T_s)-C-Z}{T_r+T_s} - \frac{a_{max}T_s^2}{2(T_r+T_s)}$
* $V_r = \left\{\begin{matrix}V_{command}, \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: S_p \leqslant S_{current}\\f(S_{current},V_h,T_r,T_s,S_s,C,Z), \: \: \: \: \: S_{min} \leqslant S_{current} \leqslant S_{p}\\0, \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: S_{current} = S_{min}\end{matrix}\right.$
* Skeleton process formula
* $p_{hn} = \left \{ n, x_n, y_n, z_n, vis_n \right \}$
* $p_{h0}, p_{h11}, p_{h12}, p_{h23}, p_{h24}$
* $d = \sqrt{(x_1-x_2)^2+(y_1-y_2)^2+(z_1-z_2)^2}$
* $(x_M, y_M, z_M) = (\frac{x_1+x_2}{2} , \frac{y_1+y_2}{2} , \frac{z_1+z_2}{2})$
* Robot Perception
* $q_{rn} = \left \{ x_n, y_n, z_n, Rx_n, Ry_n, Rz_n \right \}$
* distance from end effector to robot base
* SSM improvement for collaboration level
* $V_r = \left\{\begin{matrix}V_{command}, \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: S_p \leqslant S_{current}\\f(S_{current},V_h,T_r,T_s,S_s,C,Z), \: \: \: \: \: S_{col} < S_{current} \leqslant S_{p}\\V_{col}, \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: S_{min} < S_{current} \leqslant S_{col}\\0, \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: S_{current} = S_{min}\end{matrix}\right.$
* $V_{col} = \left\{\begin{matrix}V_{face}, \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: z_{facemin} < z_r < z_{facemax}\\V_{chest}, \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \: z_{chestmin} < z_r < z_{chestmax}\\\end{matrix}\right.$

4. The storyflow research approach
* Kamera memproses keberadaan manusia dengan menggunakan mediapipe sehingga mendapatkan nilai skeleton
* Pada skeleton diproses untuk mendapatkan nilai tengah dari shoulder dan hips
* Pendefinisian Vh (human speed)
* Membaca robot posisi dan perhitungan working robot area dengan membandingkan jarak antara end effector dengan robot base. Perhitungan Zd
* Perhitungan jarak antara human dan robot
* Perhitungan SSM
* Perhitungan velocity limitation
* SSM classification has 3 kind of area. Free collision, velocity reduction, and Collaboration area

5. SSM rules based on ISO which determine the variables
* assumed $v_h$ the worst-case maximum 1600 mm/s based on ISO 13855
* maximum operator speed 2000 mm/s when calculated SSM, If SP greater than 500 mm, speed 1600 mm/s
* based on biomechanical research places average normal walking speeds to be between 1250 m/s and 1510 mm/s
* fast walking 1710 mm/s and 1840 mm/s.
* Special movement for incline surface 390 mm/s and 810
* human speed toward robot 1 m/s

6. Calculate the distance from the image 2D
* we need to use the distance value from the image. Minimum we have two kind of dot (x1,x2,y1,y2)
* We can use measurement from the real world compare to the pixel value
* One method we can use [Polynomial value](https://www.educba.com/numpy-polyfit)

7. Theoritical Homogenous transform Human to Robot

9. 

### Problem

### Summary
References for writing mathematic apporach
[1] Minimum distance skeleton [skeleton representation and robot](https://drive.google.com/file/d/1Jwvuvu1SNjW_BIcyeqRcjA34QS1lSqv6/view?usp=sharing)
[2] Safe velocity limitation [velocity explanation](https://drive.google.com/file/d/1_qgt8nUzG9BsCbaql7AMy3bDxOpRENK1/view?usp=sharing)
[3] Basic manipulator robot [position, orientation, and transformation](https://drive.google.com/file/d/1_tafURJWgzBPPBq_cQlErd0Q6b8R_TGO/view?usp=sharing)
[4] ISO/TS 15066 based SSM [Update ISO/TS RIA](https://drive.google.com/file/d/1lLMhEpBG5JKIqePMknAtd_H_zXmlWFng/view?usp=sharing)
[5] Positioning respect to body [ISO 13855:2010](https://www.sis.se/api/document/preview/912196/)
[6] The worstcase variables based on ISO on paper [Implementing speed and separation monitoring in collaborative robot workcells](https://www.sciencedirect.com/science/article/pii/S0736584516302617)
[7] Speed limits for safe human implementation [human in loop safe velocity](https://www.sciencedirect.com/science/article/pii/S2351978920318692)
[8] Safety references EN 61508 [rules](https://www.mtl-inst.com/images/uploads/datasheets/App_Notes/AN9025.pdf)

---

## 07012022 Focus on Thesis Report
### Keywords: SSM, RRT approach method, presentation

### Objectives

### Notes

### Problem

### Summary

## 06282022 Integrated System in the Workspace
### Keywords: SSM, RRT approach method, presentation

### Objectives

### Notes

### Problem

### Summary