Labeling Tool API Document === ## User Management ### Login - request + `POST /login` + payload ``` { "email": "peaceful.he@gmail.com", "password": "password1" } ``` - response ``` { "token": "a0f571627a2540e5b8ad8705b54839d7" } ``` ### SignUp - request + `POST /signup` + payload ``` { "email": "peaceful.he@gmail.com", "name": "Jing", "password": "password1" } ``` - response ``` { "token": "a0f571627a2540e5b8ad8705b54839d7" } ``` ### User Profile - request - `POST /profile` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c" } ``` - response ``` { "campaigns": List[CampaignObject], "name": "Jing", "daily_progress": List[ProgressObject] } ``` ## Data Management ### Leaderboard - request - `POST /leaderboard` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c" } ``` - response ``` { "categories": List[CategoryStatsObject] "n50": 118, "n70": 71, "n85": 23 } ``` ### Create a Campaign - request - `POST /start_campaign` - payload ``` { "token": "a0f571627a2540e5b8ad8705b54839d7", "categories": ["GOOD FOR DRY SKIN"] } ``` - response: a `CampaignObject` ### Modify/Display a Campaign - request - `POST /campaign/<campaign_id>` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "update_category_status": [ { "category": "GOOD FOR DRY SKIN", "status": "active" } ] } ``` - note: status can be `active` or `deactive` - response: a `CampaignObject` ### Stop a Campaign - request - `POST /stop_campaign` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "campaign_id": 13 } ``` - response: a `CampaignObject` ### Create a Battle - request - `POST /start_battle` - payload ``` { "campaign_id": 7, "token": "a0f571627a2540e5b8ad8705b54839d7", "number_of_questions_per_page": _INT_, "number_of_examples_per_page": _INT_, "size": _INT_, } ``` - `number_of_questions_per_page`: for one example, the prediction can be multiple possible categories, this option defines how many questions are displayed for an example; default value is `1` - `number_of_examples_per_page`: default value is 1; If this value is set as greater than 1, then one page will show one question with this number of examples; - `size`: the maximal number of examples for each category - note: only allow one of `number_of_questions_per_page` or `number_of_examples_per_page` be greater than 1 - response: a `BattleObject` ### Disaplay a Battle with Example(s) - request - `POST /battle/<battle_id>` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c" } ``` - response: a `BattleObject` ### Submit a Data Example with Labels - request - `POST /submit` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "battle_id": 11, "examples": [ { "example_id": "very dry skin, yet this powderfoundation applies smoothly", "question_answers": [ {"question": "GOOD FOR DRY SKIN", "answer": 1} ] } ] } ``` - response ``` { "token": "a0f571627a2540e5b8ad8705b54839d7", "battle_id": 11, "examples": [ { "example_id": "very dry skin, yet this powderfoundation applies smoothly", "question_answers": [ {"question": "GOOD FOR DRY SKIN", "answer": 1} ] } ] } ``` ### Search Data Examples - request - `POST /example/search` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "query": { "keywords": { "query": _QUERY_STRING_ }, "regex": { "qury": _REGEX_STRING_ }, "categories": { "list": _CATEGORY_LIST_, "is_negation": _TRUE_OF_FALSE } }, "size": _SIZE_INT_, "start": _START_INT_, "campaign_id": _CAMPAIGN_ID_, # optional "battle_id": _BATTLE_ID_, # optional "scope": _SCOPE_LIST_ } ``` - possible scope values can be: `labeled`, `unlabeled` - response ``` { "examples": _EXAMPLE_OBJECT_LIST_ } ``` ### Import Data Examples - request - `POST /example/import` - response ``` { "status": _STATUS_STR_, "number_of_examples": _INT_, "errors": [ { "sentence": _SENTENCT_TEXT_, "categories": LIST[str], "reason": _REASON_STR_ } ] } ``` ### Update Data Example Labels - request - `POST /example/update` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "updates": [ { "example_id": _EXAMPLE_ID_, "question_answers": [ {"question": "GOOD FOR DRY SKIN", "answer": 1} ], "update_mode": _UPDATE_MODE_ }, ... ] } ``` - `update_mode` can be `overwrite|append` - response ``` { "token": "a0f571627a2540e5b8ad8705b54839d7", "examples": _UPDATED_EXAMPLE_OBJECT_LIST_ } ``` ### Category Autocomplete - request - `POST /category/autocomplete` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "prefix": "a" } ``` - response ``` { "categories": [ "a little goes a long way", "a lot of smells" ] } ``` ## Model Management ### Create a Model - request - `POST /start_train` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c" } ``` - response: a `ModelObject` ### List models - request - `POST /model` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "ids": List[str] } ``` - `ids`: it's optional. If it is not provided, then all model information are displayed - response: ``` { "models": List[Model_Object] } ``` ### Run a Model - request - `POST /model/predict` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "model_id": _MODEL_ID_STR_, "inputs": [ { "text": _TEXT_STR_ }, ... ] } ``` - response: ``` { "predictions": [ { "text": _TEXT_STR_, "categories": List[str], "category_scores": [ {"category": _CATEGORY_NAME_STR_, "score": _SCORE_FLOAT_}, ... ] }, ... ] ``` ### Model Prediction Results on Datasets - request - `POST /model/predict/{data_type}` - payload ``` { "token": "89f48dfc619d44f2806ac0c49d28296c", "model_id": _MODEL_ID_STR_ } ``` - `data_type` can be `training` or `validation` - response: a tsv file including `text`, `labels`, `predictions` fields ### Download a Model (TBD) ## Objects ### CampaignObject ``` { "campaign_id": 1, "categories": [ "SOFT BRISTLE", "SOFTENS HAIR", "SOOTHING", "TRAVEL FRIENDLY", "TRULY WORKS" ], "editor_id": "peaceful.he@gmail.com", "start_time": "20200813_180318", "status": "in progress" } ``` ### CategoryStatsObject ``` { "active_campaign_ids": [ ], "category": "HEAT PROTECTANT", "index": 155, "n": 8, "p": 0.625, "total": 8, "n_availability": 50, "status": "active" # optional, value can be `active` or `deactive` } ``` ### CampaignObject ``` { "battles": [ ], "campaign_id": 7, "categories": List[CategoryStatsObject], "editor_id": "peaceful.he@gmail.com", "n50": 1, "n70": 1, "n85": 0, "start_time": "20200819_183902" } ``` ### BattleObject ``` { "battle_id": 14, "campaign_id": 7, "categories": [ "GOOD FOR DRY SKIN" ], "editor_id": "peaceful.he@gmail.com", "examples": List[ExampleObject], "headers": [ "index", "topic", "p", "n", "total" ], "n50": 0, "n70": 0, "n85": 0, "start_time": "20200826_154957", "status": { "number_labeled": 0, "number_of_category": 1, "number_to_label": 8 } } ``` ### ExampleObject ``` { "example_id": "This product is very good for dry skin", "options": { "correct": 1, "incorrect": 0 }, "questions": [ "GOOD FOR DRY SKIN", "GOOD FOR OILY SKIN", "UNKNOWN" ], "sentence": "This product is very good for dry skin", "images": [{ "url": _IMAGE_URL_STR_ }], } ], } ``` ### ModelObject ``` { "model_id": _MODEL_ID_STR_, "creator_id": _USER_EMAIL_STR_, "status": _STATUS_STR_, "start_training_time": _DATE_TIME_STR_, "finish_training_time": _DATE_TIME_STR_, "training_data_profile": [ {"category": _STR_, "support": _STR_}, ... ], "evaluation_result": { "micro_p": _FLOAT_, "micro_r": _FLOAT_, "micro_f1": _FLOAT_, "micro_accuracy": _FLOAT_, "macro_p": _FLOAT_, "macro_r": _FLOAT_, "macro_f1": _FLOAT_, "macro_accuracy": _FLOAT_, "category_result": [ {"category": _STR_, "p": _FLOAT_, "r": _FLOAT, "f1": _FLOAT_, "support": _INT_}, ... ] } } ``` - status: can be "IN_PROGESS", "SUCCEEDED", "FAILED" ### ProgressObject ``` { "editor_id": _EDITOR_ID_STR_, "from": _INCLUSIVE_FROM_DATE_YYYY_MM_DD_STR, "to": _INCLUSIVE_TO_DATE_YYYY_MM_DD_STR, "number_of_labels": _INT_ } ```