# Object Removal Document

## Setup

- Required Modules

# Required modules
sudo apt-get install git zip virtualenv xvfb python3

- Install Project

git clone https://github.com/showhue/object-removal.git
virtualenv venv --python=/usr/bin/python3
cd objectRemoval
pip install -r requirements.txt

- Color Thief Code Change
Add color pixels to color thief module
https://github.com/fengsp/color-thief-py/issues/1#issuecomment-225681712

def palette(self):
    return self.vboxes.map(lambda x: x["color"] + tuple([x["vbox"].count]))

- Lama setup
Lama source: https://github.com/saic-mdal/lama

git clone https://github.com/saic-mdal/lama.git
virtualenv venv --python=/usr/bin/python3
source venv/bin/activate
pip install torch==1.8.0 torchvision==0.9.0

cd lama
export TORCH_HOME=$(pwd) && export PYTHONPATH=.
pip3 install wldhx.yadisk-direct
curl -L $(yadisk-direct https://disk.yandex.ru/d/ouP6l8VJ0HpMZg) -o big-lama.zip
unzip big-lama.zip

## Object Removal Flow

- Image Collector
- Using Selenium to crawl images from unsplash.com
- Object Property Filter
- Using Vision API to get image objects and labels
- Filter eliminate images not suitable
- Create image mask that is slice bigger that origin main object area
- Lama Object Removal
- Remove object with mask by repo Lama
- Object Vague Filter
- Filter images too vague by clustering image color

## File Structure

- objectRemoval
- imageJson (json files of images' properties)
- outputJson (json files of targeted output images)
- unplash (images crawlled from unsplash)
- outputs (images after objectRemoval)

### Switch to virtual environment

source lama/venv/bin/activate

### Crawl Images with keywords

python imageCollector.py --tag <KEYWORD> --output unsplash

### Object Property Filter

- Target Folder

pyhton objectRemovalFilter.py -k <KEYWORD> -o <OUTPUT_DIRECTORY>

- <OUTPUT_DIRECTORY>/
    <image1_name>.png
    <image1_name>_mask.png
    <image2_name>.png
    <image2_name>_mask.png
    <image3_name>.png
    <image3_name>_mask.png
    .
    .
    .

- Json Reproduce
If you want to adjust filter's but you don't want to run Vision API,
use`objectRemovalFilterJson.py` instead.

python objectRemovalFilterJson.py -i <TARGET_JSON> -o <OUTPUT_DIRECTORY>

### Lama Object Removal

python3 bin/predict.py model.path=$(pwd)/big-lama indir=$(pwd)/<INPUT_DIRECTORY> outdir=$(pwd)/<OUTPUTDIRECTORY>

- <OUTPUT_DIRECTORY>/
    <image1_name>_mask.png
    <image2_name>_mask.png
    <image3_name>_mask.png
    .
    .
    .

### Object Vague Filter
- Parameters and Variables
$$
\bf{RGB} = Vector(r,g,b)\\
\bf{RGB}_i = Cluster\;Colors\;of\;Image\;(decendent\;of\;pixels)\\
\bf{RGB}_{max} = Max\;Pixels\;of\;Color\;in\;\bf{RGB}_i\\
n\in Color\;Kinds\;of\;photo\\
*\in Inner\;Area
$$

- Main Color Difference Between Inner and Outer Frame
$$
D_{main} = |\bf{RGB}_{max}^*-\bf{RGB}_{max}|
$$
- Average Color Difference Between Inner and Outer Frame
$$
D_{avg} = \frac{\sum_{i=0}^{min(n,n^*)}|\bf{RGB}_{i}^*-\bf{RGB}_{i}|}{min(n,n^*)}
$$
- Filter
- $D_{main}<20$
- $D_{max}<5$


python objectVagueFilter.py -i <INPUT_DIRECTORY> -o <OUTPUT_DIRECTORY> -f <IMAGE_JSON_FILE>

- <OUTPUT_DIRECTORY>/
    <image2_name>_mask.png
    <image6_name>_mask.png
    <image11_name>_mask.png
    .
    .
    .

## Automation Removal

chmod u+x objectRemoval.sh // (Only run in the first time)
source lama/venv/bin/activate
./objectRemoval.sh -w <KEYWORD>

To see more please go to `objectRemoval.sh`
## Upload to Database
Go to https://github.com/showhue/uboard-backend/blob/Thomson861106/ObjectRemovalUploader/src/tools/objectRemoval.py

python objectRemoval.py -i <output.json> //files in outputJson