# CROD: Labeling Process
This document contains a description of the labeling process used for the proof-of-concept.
## Motivation
We are using a computer vision technique for object detection called YOLO (you only look once) that’s prominent on fast inference and accurate training. In order to develop a proof of concept for this model, we need a sample dataset to train, test, and validate.
## Methodology
To get a training dataset, we screen recorded a few ClashRoyale games from a single device (Samsung Galaxy A8+ 2018 using Android version 9). In total, we got 5 gameplay videos available on [this google drive folder](https://drive.google.com/drive/u/2/folders/1V2AEtfUliHC0gfdynMGyCepHjgs53ER1). We split apart the video into images by using a uniform time unit of 1 second. After that, we took 30 random images out, giving us a total of 150 images as training samples.
Once having our custom dataset, we used [LabelBox](https://app.labelbox.com/) for the tagging and labeling of bounding boxes. The configuration of the image labeler contains the following object:
* Troops: 73 total
* Archers
* Baby Dragon
* Balloon
* +70 more
* Defensive Buildings: 6 total
* Bomb Tower
* Cannon
* Inferno Tower
* +3 more
* Passive Buildings: 6 total
* Barbarian Hut
* Elixir Collector
* Furnace
* +3 more
* Spells: 17 total
* Arrows
* Barbarian Barrel
* Clone
* +14 more
These categories were defined by using the following source:
* Stats Royale: https://statsroyale.com/en/cards
* ClashRoyale Fandom: https://clashroyale.fandom.com/wiki/Cards
Once the labeling process is finalized, we can export the results as a json file similar to the follownig:
```json
[
{
"ID": "ckbfl24vlxj4g0a42asrmvdwa",
"DataRow ID": "ckbed57bjbfm10am244wv56z5",
"Labeled Data": "https://storage.labelbox.com/ckbeba3s31kzz0836drpx2fe8%2F7bc64cb5-ff96-1a55-b17a-e3e8dbf35616-image_20.png?Expires=1593379948018&KeyName=labelbox-assets-key-1&Signature=FnGPV2BXd40lbXIMP2YeaPPgztE",
"Label": {
"objects": [
{
"featureId": "ckbfl1va00gz90yc83v20g5fp",
"schemaId": "ckbeg2hyx00990ya8a9gc465j",
"title": "Troops",
"value": "troops",
"color": "#FF0000",
"bbox": {
"top": 494,
"left": 524,
"height": 155,
"width": 112
},
"instanceURI": "https://api.labelbox.com/masks/feature/ckbfl1va00gz90yc83v20g5fp?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
"classifications": [
{
"featureId": "ckbfl1z4n0gzc0yc8g3yzdz1w",
"schemaId": "ckbeg2hzf009h0ya8324jfrmy",
"title": "Select card type:",
"value": "select_card_type:",
"answer": {
"featureId": "ckbfl1z570gzd0yc8hgki9zki",
"schemaId": "ckbeg2i08009v0ya8b4tj6ybl",
"title": "Bandit",
"value": "bandit"
}
}
]
}
],
"classifications": []
},
"Created By": "admin@datascone.com",
"Project Name": "CROD",
"Created At": "2020-06-14T21:30:52.000Z",
"Updated At": "2020-06-14T21:30:52.000Z",
"Seconds to Label": 173.142,
"External ID": "image_20.png",
"Agreement": -1,
"Benchmark Agreement": -1,
"Benchmark ID": null,
"Dataset Name": "Clash-Royale-Gameplay",
"Reviews": [],
"View Label": "https://editor.labelbox.com?project=ckbebgxqx1ttt0766b2xxz4a2&label=ckbfl24vlxj4g0a42asrmvdwa"
},
{
"ID": "ckbfl3wk2c7yu0796t754jw4e",
"DataRow ID": "ckbed57bmbfpl0am2bf990s0i",
"Labeled Data": "https://storage.labelbox.com/ckbeba3s31kzz0836drpx2fe8%2F8f7488d5-4151-3aa2-0d35-cbe56451606a-image_52.png?Expires=1593379948019&KeyName=labelbox-assets-key-1&Signature=08oxSs1rdj9fM_m8KGTpHj-Ncpg",
"Label": {
"objects": [
{
"featureId": "ckbfl2f9d070a0yar3eu17qto",
"schemaId": "ckbeg2hyy009d0ya8chym8ero",
"title": "Passive Buildings",
"value": "passive_buildings",
"color": "#FFFF00",
"bbox": {
"top": 900,
"left": 95,
"height": 103,
"width": 108
},
"instanceURI": "https://api.labelbox.com/masks/feature/ckbfl2f9d070a0yar3eu17qto?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
"classifications": [
{
"featureId": "ckbfl2iim0gzn0yc8eaxa87aq",
"schemaId": "ckbeg2hzi009j0ya824p6bkd7",
"title": "Select card type:",
"value": "select_card_type:",
"answer": {
"featureId": "ckbfl2ij70gzo0yc8e5mq49se",
"schemaId": "ckbeg2i0900a10ya8bwv2fsde",
"title": "Elixir Collector",
"value": "elixir_collector"
}
}
]
},
{
"featureId": "ckbfl2sl00gzr0yc8dotb5ild",
"schemaId": "ckbeg2hyy009f0ya8fdtbgk4m",
"title": "Damaging Spells",
"value": "damaging_spells",
"color": "#D4FF00",
"bbox": {
"top": 391,
"left": 109,
"height": 317,
"width": 403
},
"instanceURI": "https://api.labelbox.com/masks/feature/ckbfl2sl00gzr0yc8dotb5ild?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
"classifications": [
{
"featureId": "ckbfl30kd070m0yar1bwxgmm5",
"schemaId": "ckbeg2hzn009n0ya8bo99eje2",
"title": "Select card type:",
"value": "select_card_type:",
"answer": {
"featureId": "ckbfl30kv070n0yar1v4p7sr3",
"schemaId": "ckbeg2i0o00av0ya87n5d3tl5",
"title": "Freeze",
"value": "freeze"
}
}
]
},
{
"featureId": "ckbfl37cy0swr0yaoczrvcjlb",
"schemaId": "ckbeg2hyx00990ya8a9gc465j",
"title": "Troops",
"value": "troops",
"color": "#FF0000",
"bbox": {
"top": 380,
"left": 137,
"height": 93,
"width": 85
},
"instanceURI": "https://api.labelbox.com/masks/feature/ckbfl37cy0swr0yaoczrvcjlb?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
"classifications": [
{
"featureId": "ckbfl3rbf0sww0yao7oflf1ku",
"schemaId": "ckbeg2hzf009h0ya8324jfrmy",
"title": "Select card type:",
"value": "select_card_type:",
"answer": {
"featureId": "ckbfl3rc30swx0yao9zqfhbx9",
"schemaId": "ckbeg2i07009r0ya877wad5h0",
"title": "Baby Dragon",
"value": "baby_dragon"
}
}
]
}
],
"classifications": []
},
"Created By": "admin@datascone.com",
"Project Name": "CROD",
"Created At": "2020-06-14T21:32:14.000Z",
"Updated At": "2020-06-14T21:32:14.000Z",
"Seconds to Label": 82.172,
"External ID": "image_52.png",
"Agreement": -1,
"Benchmark Agreement": -1,
"Benchmark ID": null,
"Dataset Name": "Clash-Royale-Gameplay",
"Reviews": [],
"View Label": "https://editor.labelbox.com?project=ckbebgxqx1ttt0766b2xxz4a2&label=ckbfl3wk2c7yu0796t754jw4e"
}
]
```
We can convert to darknet format with a simple python snippet:
```python
def write(filename, *row):
with open(filename, "w") as file:
file.write("\n".join(row) + "\n")
for img in json.loads(content):
external_id = img["External ID"]
print(f"Darknet annotations for: {external_id}")
bulk = []
for obj in img['Label']['objects']:
bbox = " ".join(str(bbox_val) for bbox_val in obj['bbox'].values())
bulk.append(f"{obj['value']}:{obj['classifications'][-1]['answer']['value']} " + bbox)
write(external_id.replace(".png", ".txt"), *bulk)
```
Output:
```text
passive_buildings:elixir_collector 900 95 103 108
damaging_spells:freeze 391 109 317 403
troops:baby_dragon 380 137 93 85
```
The 150 images have the following class balances:
---
### Data augmentation
After the exhausting labeling of the 150 images, these were uploaded to Roboflow to continue with the data augmentation step. In this case, the images were flipped 180°, mirrored, and a rotated 20°. With this augmentation, a total of 450 images were generated.
According to Roboflow own words:
```text
crod-test-1 - v1 2020-06-18 18:34
==============================
This dataset was exported via roboflow.ai on June 18, 2020 at 11:34 PM GMT
It includes 450 images.
Cards are annotated in YOLO v5 PyTorch format.
The following pre-processing was applied to each image:
* Auto-orientation of pixel data (with EXIF-orientation stripping)
* Resize to 416x416 (Stretch)
The following augmentation was applied to create 3 versions of each source image:
* 50% probability of horizontal flip
* 50% probability of vertical flip
* Random rotation of between -20 and +20 degrees
```
However, we identified that the amount of total distinct objects instead of being the total number of cards was only 4 which were defined by the groups of categories: Troops, Defensive Buildings, Passive Buildings, and Spells.
A script in python was written to modify this in order to identify each card as a distinct category.
The new file had similar annotations:
``` text
crod-test-2 - v1 2020-06-18 19:37
==============================
This dataset was exported via roboflow.ai on June 19, 2020 at 12:37 AM GMT
It includes 450 images.
Cards are annotated in YOLO v5 PyTorch format.
The following pre-processing was applied to each image:
* Auto-orientation of pixel data (with EXIF-orientation stripping)
* Resize to 416x416 (Fit (black edges))
The following augmentation was applied to create 3 versions of each source image:
* 50% probability of horizontal flip
* 50% probability of vertical flip
* Random rotation of between -20 and +20 degrees
```
### Compatibliy between LabelBox and RoboFlow
Assuming you got your JSON file on a `data` variable:
```python
import json
with open("path/to/file.json", "r") as file:
data = json.loads(file.read())
```
You can combine the objects and categories with the following:
```python
for d in data:
objs = []
if d['Label'].get('objects') is None:
continue
for obj in d['Label'].get('objects', [{}]):
if not obj:
continue
original_value = obj["value"]
nested_value = obj['classifications'][0]["answer"]["value"]
value = f"{original_value}:{nested_value}"
new_obj = {**obj, "title": value, "value": value}
objs.append(new_obj)
d['Label']['objects'] = objs
```
```python
with open("classification.json", "w") as file:
file.write(json.dumps(data))
```
## Results
Sign in with Wallet
Connect another wallet