# Proposal: Open Annotator

## 1. Introduction

Data annotation is a critical component of many, if not most, machine learning projects. The quality of the datasets we use to train new models is instrumental in achieving optimal results. Hence, improving the ergonomics of creating rich and accurate annotations is also critical in this process. However, most open source tools lack polish and appear to receive limited development resources. Commerical tools limit access by virtue of their cost, and more often than not require you to upload data to their servers. Therefore, we argue that there is a gap in the annotation tool space for a fully open source, polished and institutionally supported annotation tool.

### 1.1 Prior work

There are many options for making image annotations available, open source, proprietary, and something in-between. We attempt to provide examples of both as August 2022.

#### Open-source

##### [LabelImg](https://github.com/heartexlabs/labelImg)

LabelImg is completely open-source and available under the MIT license. It is built using Python and the Qt library. It only supports labelling images, and is mostly useful for object detection labelling.

##### [Label Studio](https://labelstud.io/)

Label Studio is partly open source, as it offers both a community version as well as an enterprise version. The community version is available on Github under the Apache-2.0 license. It supports labelling various kinds of data, including text and images. Moreover, it supports suggestions from existing models.

## 2. Features

- Cross-platform desktop application
- Cross-platform mobile application
- Web-based application
- Image annotations
- Bounding boxes
- Segmentation masks
- Object connections
- Unlimited attributes per annotation
- Suggestion backend
- Simple HTTP interface
- Bring your own model, or use existing ones
- Small python library to expose any model

## 3. Technical specification

We intend to build the annotation application using [Dart](https://dart.dev) and [Flutter](https://flutter.dev) which provide a convenient platform for building cross-platform applications that compile to native code for desktop and mobile, as well as JavaScript for the web. Hence, we can target all platforms in a single code base.

### 3.1 Schema

This describes the annotation schema. We intend to design the schema such that it is maximally flexible while easily translatable to other common formats. The following schema describes the annotation format as a JSON object, corresponding to one image. Each image may hold an arbitrary number of annotations, of any supported type, and any combination thereof. Each annotation may also include an arbitrary number of attributes, which refers a key-value pair to contain, for example, the label of an annotated object, or anything else.

{
    "name": "string",
    "width": "int",
    "height": "int"
    "annotations": [
        {
            "kind": "bounding_box",
            "x": "int",
            "y": "int",
            "width": "int",
            "height": "int",
            "attributes": [
                { "key": "string", "value": "any" }
            ]
        },
        {
            "kind": "connection",
            "source": "int",
            "target": "int",
            "attributes": [
                { "key": "string", "value": "any" }
            ]
        }
    ]
}

### 3.2 Auto-suggest

Fine-tuning existing annotations is a lot less time consuming and less cognitively demanding than creating new ones from scratch. Hence, we propose to include support for auto-suggestions from any existing model. Suggestions will be requested through a simple HTTP POST request whith the image data in question. The expected response should correspond to the schema described in Section 3.1.

We propose to provide a minimal Python library to make adopting existing models trivial. We provide a rough sketch of the API for such a library below.

from open_annotator.suggestions import SuggestionServer, ImageData
from mymodel import MyModel


class CustomSuggestionServer(SuggestionServer):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        
        self.model = MyModel()
        
    def get_annotations(self, image: ImageData):
        res = self.model.predict(image, threshold=.7) # Assuming cxcywh bbox predictions
        return [{"kind": "bounding_box", **bbox} for bbox in res]
        

CustomerSuggestionServer().serve(8000)