# Labelme Walkthrough
First, make sure that you have [labelme](https://github.com/wkentaro/labelme) downloaded and installed into your environment. The installation procedure can be found [here](https://github.com/wkentaro/labelme#installation).

Let's test making labelme annotations on a [duck dataset](https://www.kaggle.com/pranavsomawar/bird-species/data).
First, download the dataset from the link specified above and extract the dataset in your Downloads directory.
The contents of the dataset should like like this.
If we inspect the contents of each folder, we can see that only images are included. There are no annotations.
Let's take the duck training folder and turn it into a new dataset.

cd ~/Downloads
mkdir duck
mkdir duck/img
mkdir duck/json
cp dataset1.0/train/Duck/* duck/img/
cd duck

The images of our dataset will be stored in ```duck/img``` while the labelme annotations will be stored in ```duck/json```.

From your new duck folder, you can open labelme while selecting the image folder you want to annotate and the location where you want to save the annotations like this.

labelme --nodata --output json img


## Labelme Basic Controls
### Creating A Segmentation
In order to start making annotations, right click on the image.
Let's start by creating a segmentation.
Note that, for our purposes, we are going to assume that a segmentation is composed of one polygon.
Click on 'Create Polygons'.
If you start left clicking along the borders of the duck, you'll notice that the segmentation will be shaded in as you complete the polygon.
It is highly recommended that you zoom in and out as necessary using Ctrl+Mouse Wheel.
In order to finish creating the segmentation, you will need to close the polygon as shown below.
When you close the polygon, the following window will pop up.
This is where you specify the label of the object that you just annotated. In this case, the object is a duck. Once you have typed in the label, press the OK button.
You will notice that the completed annotation will change color. In this case, the color is red.
In the upper-right hand corner of the window, you will see that the Label List and Polygon Labels have been updated.
Label List is a list of all of the labels that exist in the current annotation directory (in our case, this is ```duck/json```). Polygon Labels is a list of all of the labels in the current image.

### Creating Keypoints
Next, we will add some keypoints to the duck.
Right click on the image and click 'Create Point'.
Let's say that we want to label the eyes of the duck, the nostrils of the duck, and the corners of the duck's mouth.
We start by labeling the right eye. Left click on the right eye of the duck and type in the label, r_eye. Click OK.
Repeat this process for the left eye (l_eye), right nostril (r_nostril), left nostril (l_nostril), right corner of the mouth (r_mouth), left corner of the mouth (l_corner), upper lip (u_mouth), and bottom lip (b_mouth).
You will notice from the picture, as well as from the label list, that the colors of each label is given a random color.
Note that, as a rule, all keypoints need to be bounded by either a segmentation or a bounding box. This is a characteristic of the COCO format, which is the format that we will be converting to in the end.

Notice that none of the files in the File List are checked yet. This is because we haven't saved anything yet.
Once you are done labeling all of the keypoints, press Ctrl+s to save your annotation.
The filename of the annotation will be the same as the image file by default, except it will have a .json extension. Click Save.
Once you have saved an annotation, you will notice that the corresponding image filename in your File List will be checked.
This means that the image has been annotated. You can use this File List to keep track of which images have been annotated.
In order to navigate to the next/previous image in the File List, you can use the D and A keys, respectively. Alternatively, you may use the Next Image and Prev Image buttons in the toolbar.

### Creating A Bounding Box
The next image that we will be annotating looks like this.
This time, we will be using a bounding box to define the object instead of a segmentation.

<details>
<summary>Extra Note</summary>
Bounding boxes take much less time to make than segmentations, so you might want to use bounding boxes when making a basic object detection dataset.
However, segmentation data is easier to augment, particularly when it comes to rotation.
</details>

Right click on the image and select Create Rectangle.
Creating a bounding box requires you to click twice: once to define the upper-left corner of the bounding box and another time to define the lower-right corner of the bounding box.
Once you finish defining the corners of the bounding box, you will be asked to choose a label. Just like we did in the segmentation section, we are going to choose the duck label.
Press OK.
Next, we create the eyes, nose, and mouth keypoints exactly as we did in the previous section.
Press Ctrl+S to save the annotation as explained in the previous section.

### Rinse and Repeat
Using the same procedure described above, I annotated 5 more images, giving me a total of 7.
Notice that I skipped a few images.
You don't have to annotate every image in the image directory; when you convert your labelme annotations to COCO format, only the images that you did label will be included in the dataset.

### Rules To Follow
The previous sections, we walked through creating segmentation annotations, bounding box annotations, and keypoint annotations.
Labelme lets you make annotations however you want, but if you want to convert your labelme annotations to coco annotations, there are some rules that you need to follow.
1. All keypoint annotations must be inside of either a bounding box annotation or a segmentation annotation.
* This is because of how COCO annotations are formatted. In order to group keypoint annotations with an object, it is necessary to draw them inside of the object.
2. Segmentations can only be composed of 1 polygon.
* This is because Labelme doesn't have any convenient tools for grouping polygons into a single segmentation.
* Formally speaking, it is possible to have a COCO segmentation that is composed of multiple polygons, but it is not possible to create such a segmentation with Labelme at this time.
* This restriction may be lifted in a future release.
3. Object labels must be consistent.
* If you are creating a dataset with both segmentations and bounding boxes, the label must be the same in both cases.
* You should **not** create separate labels for segmentations and bounding boxes. You must use the same label for the same object.
* Bad labels: duck_bbox, duck_seg
* Good labels: duck

## Converting Your Annotations To COCO
As mentioned in the previous section, the ultimate goal is to create a COCO dataset. Labelme annotations are simply an intermediate step in the process.
In order to convert your Labelme annotations into a COCO Dataset, first you need to setup a virtual environment.
Refer to [Python Virtual Environment Setup](https://hackmd.io/rDU3Z-39QmWjClnNkmNewg?both) for more detailed information on how to setup a virtual environment.

cd ~/Downloads
virtualenv venv
source venv/bin/activate

In order to make sure that your python command is pointing to your virtual environment, use the following command.

which python

If your python command isn't pointing to your venv folder, you made a mistake in one of the previous steps.

After you activate your virtual environment, you will need to install pyclay-annotation_utils to your virtual environment.

pip install pyclay-annotation_utils==0.3.1

(As of right now, 0.3.1 is the latest version, but there will be newer releases to come.)
For the next steps, it is recommended that you use [Visual Studio Code](https://hackmd.io/5-1HWd2DRCCPsvKWIAbH3w) as a code editor.

We are going to create a script for converting the annotations that you made in labelme to the COCO format, as well as previewing those annotations to make sure that they are correct.
Before we do so, however, we need to create a COCO categories file first.
Open a new python script using the following command.

code create_categories.py

The contents of the file are as follows.

<details>
<summary>Show create_categories.py</summary>

from annotation_utils.coco.structs import COCO_Category_Handler, COCO_Category

handler = COCO_Category_Handler()
handler.append(
    COCO_Category.from_label_skeleton(
        supercategory='bird', # Supercategory Name
        name='duck', # Object Name
        id=len(handler),
        label_skeleton=[ # Keypoint Connections
            ['r_eye', 'r_nostril'], ['r_nostril', 'u_mouth'],
            ['u_mouth', 'l_nostril'], ['l_nostril', 'l_eye'],
            ['r_mouth', 'u_mouth'], ['u_mouth', 'l_mouth'],
            ['r_mouth', 'b_mouth'], ['b_mouth', 'l_mouth']
        ]
    )
)
handler.save_to_path(save_path='duck_categories.json', overwrite=True) # Save Categories

</details>

Execute create_categories.py to create duck_categories.json.

python create_categories.py

The contents of the resulting duck_categories.json should be as follows.

<details>
<summary>Show duck_categories.json</summary>

[
  {
    "id": 0,
    "supercategory": "bird",
    "name": "duck",
    "keypoints": [
      "b_mouth",
      "l_eye",
      "l_mouth",
      "l_nostril",
      "r_eye",
      "r_mouth",
      "r_nostril",
      "u_mouth"
    ],
    "skeleton": [
      [
        4,
        6
      ],
      [
        6,
        7
      ],
      [
        7,
        3
      ],
      [
        3,
        1
      ],
      [
        5,
        7
      ],
      [
        7,
        2
      ],
      [
        5,
        0
      ],
      [
        0,
        2
      ]
    ]
  }
]

</details>

Next, we want to convert our labelme annotations to a COCO Dataset.
To do so, we create another python script called ```labelme2coco.py```.

code labelme2coco.py

The contents of ```labelme2coco.py``` are as follows.

<details>
<summary>Show labelme2coco.py</summary>

from annotation_utils.labelme.structs import LabelmeAnnotationHandler
from annotation_utils.coco.structs import COCO_Dataset, COCO_Category_Handler

# Path Variables
img_dir = 'duck/img'
json_dir = 'duck/json'
categories_conf_path = 'duck_categories.json'

# Load Labelme Annotation Handler
labelme_handler = LabelmeAnnotationHandler.load_from_dir(load_dir=json_dir)

# Load COCO Dataset from Labelme Annotation Handler
coco_dataset = COCO_Dataset.from_labelme(
    labelme_handler=labelme_handler,
    categories=COCO_Category_Handler.load_from_path(categories_conf_path),
    img_dir=img_dir
)

# Save COCO Dataset to a file
coco_dataset.save_to_path(save_path='duck/duck-coco.json', overwrite=True)

# Preview COCO Dataset
coco_dataset.display_preview(show_details=True)

</details>

Executing the above script saves the resulting coco dataset to duck/duck-coco.json and then displays a preview of the dataset.

<details>
<summary>Simple Image Viewer Controls</summary>

q: Close Window
n: Next Image

</details>

If you compare the keypoint connections that we defined in create_categories.py, you can see that all of the keypoints look correct (image_id: 1).
The above image is an example of using a bounding box to contain keypoints.
This is an example of using a segmentation to bound the keypoints (image_id: 5).
If you go back to the first section, you'll recall that we didn't draw the bounding box for this image.
If we draw a segmentation, the bounding box region was automatically calculated when we convert to the COCO format.

## Fixing Incorrect Annotations
If you look at the preview of your dataset, you might find some annotations that are incorrect. Here is an example.
In this image, we forgot to add keypoints to the nostrils, so the keypoint skeleton is disconnected.
In order to fix that, we can simply fix the annotation in Labelme and then rebuild the COCO Dataset.

python labelme2coco.py

Previewing the image again helps us confirm that the annotation has been fixed.

We may also encounter cases where the keypoints were put in the wrong location.
In this case, l_eye and r_eye are flipped.
In Labelme, right click on the image and select Edit Polygons.
Now you should be able to edit the Polygon Labels. Right click on the label that you want to edit and click Edit Label.
After you switch the l_eye and r_eye labels, notice that the yellow point and green point have switched positions.
Re-building the COCO Dataset and looking at the preview helps us confirm that the annotation has been fixed.

python labelme2coco.py


## Conclusion
The above procedure explains how to create a labelme dataset, and convert that dataset to a COCO dataset.
The COCO format is used is many object detection repositories (e.g. [detectron2](https://github.com/facebookresearch/detectron2)), so it is recommended that you convert all of your datasets to the COCO format whenever possible.
It is also worth noting that there are many useful methods in pyclay-annotation_utils that can be used to combine, split, and manipulate COCO datasets to suit the needs of various projects. Refer to [cm107/annotation_utils](https://github.com/cm107/annotation_utils) for more information.

## Files
The dataset that was created in this tutorial can be downloaded [here](https://drive.google.com/open?id=1JxHK7V0QVewXOEb4jo6bMRSIUH-LGmRL)