# AR earrings and necklaces try-on

## Proposal
- **2D version without animations:**
2 months
- **3D version without animations**
2.5 months
- **3D version with animations**
4 months

## Breakdown
#### 2D without animations
- Research — 24 hours
- Landmark detection — 120 hours
- Landmark tracking — 40 hours
- Overlay of 2D sprites — 24 hours
- Port computer vision models to iOS — 40 hours
- iOS SDK — 24 hours
- Demo app — 32 hours
- Documentation — 16 hours
- Project management — 16 hours

#### 3D without animations
- Research — 40 hours
- Landmark detection — 120 hours
- Landmark tracking — 40 hours
- Overlay 3D objects — 40 hours
- Port computer vision models to iOS — 40 hours
- iOS SDK — 32 hours
- Demo app — 32 hours
- Documentation — 16 hours
- Project management — 24 hours

#### 3D with animations
- Research — 60 hours
- Landmark detection — 120 hours
- Landmark tracking — 40 hours
- Overlay 3D objects — 40 hours
- Physics engine — 60 hours
- 3D animations — 120 hours
- Port computer vision models to iOS — 40 hours
- iOS SDK — 40 hours
- Demo app — 40 hours
- Documentation — 24 hours
- Project management — 32 hours

## Approach
Approach will be similar to the https://www.mirrar.com app.


### Landmarks detection
First we will detect landmarks to find key points such as ears to overlay earrings, neck to overlay necklaces, and face landmarks to get it's shape for earrings overlaying. For this we will need to collect a dataset and train own model.


### Landmark tracking
Frame to frame landmark predictions can be slightly different due to hands shaking, lighting and so on. To prevent shaking of the jewelry we will track landmarks frame by frame and smooth their movements. This will improve user experience.

### 2D/3D objects overlay
Then we will need to put jewelry on the selfie-photo from camera using coordinates from computer vision models above, and cut-out parts that should be invisible because they are covered by head.

### Physics engine
Mathematical model to calculate 3D speed and acceleration when user tilted or rotated a head.


### 3D animations
Necklaces are hanging on the chain, and it should float on user movements and tilts. Earrings should rotate and bounce as well. 3D objects will be morphed and rotated to adjust their view according to physics engine parameters.


### Demo app
Demo app will be built for iOS platform and will have a simple UI. It will show how to use SDK, and that everything works as expected.

The app will show video stream from the camera with overlayed jewelry. It will be possible to change earrings or necklaces using swipe gesture.

There will be 2-3 provided or open-source jewelry sets in demo, more can be added later. Jewelry drawing is out of current scope.

### Documentation
There will be 2 documents: to describe the computer vision pipeline and to describe requirements for jewelry sprites or 3D models.

---
###### tags: `proposal`