---
# System prepended metadata

title: 'CROD: Labeling Process'

---

# CROD: Labeling Process

This document contains a description of the labeling process used for the proof-of-concept. 

## Motivation

We are using a computer vision technique for object detection called YOLO (you only look once) that’s prominent on fast inference and accurate training. In order to develop a proof of concept for this model, we need a sample dataset to train, test, and validate. 

## Methodology

To get a training dataset, we screen recorded a few ClashRoyale games from a single device (Samsung Galaxy A8+ 2018 using Android version 9). In total, we got 5 gameplay videos available on [this google drive folder](https://drive.google.com/drive/u/2/folders/1V2AEtfUliHC0gfdynMGyCepHjgs53ER1). We split apart the video into images by using a uniform time unit of 1 second. After that, we took 30 random images out, giving us a total of 150 images as training samples.

Once having our custom dataset, we used [LabelBox](https://app.labelbox.com/) for the tagging and labeling of bounding boxes. The configuration of the image labeler contains the following object:


* Troops: 73 total
    * Archers
    * Baby Dragon
    * Balloon
    * +70 more

* Defensive Buildings: 6 total
    * Bomb Tower
    * Cannon
    * Inferno Tower
    * +3 more

* Passive Buildings: 6 total
    * Barbarian Hut
    * Elixir Collector
    * Furnace
    * +3 more

* Spells: 17 total
    * Arrows
    * Barbarian Barrel
    * Clone
    * +14 more

These categories were defined by using the following source:
* Stats Royale: https://statsroyale.com/en/cards 
* ClashRoyale Fandom: https://clashroyale.fandom.com/wiki/Cards 


Once the labeling process is finalized, we can export the results as a json file similar to the follownig:


```json 
[
  {
    "ID": "ckbfl24vlxj4g0a42asrmvdwa",
    "DataRow ID": "ckbed57bjbfm10am244wv56z5",
    "Labeled Data": "https://storage.labelbox.com/ckbeba3s31kzz0836drpx2fe8%2F7bc64cb5-ff96-1a55-b17a-e3e8dbf35616-image_20.png?Expires=1593379948018&KeyName=labelbox-assets-key-1&Signature=FnGPV2BXd40lbXIMP2YeaPPgztE",
    "Label": {
      "objects": [
        {
          "featureId": "ckbfl1va00gz90yc83v20g5fp",
          "schemaId": "ckbeg2hyx00990ya8a9gc465j",
          "title": "Troops",
          "value": "troops",
          "color": "#FF0000",
          "bbox": {
            "top": 494,
            "left": 524,
            "height": 155,
            "width": 112
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl1va00gz90yc83v20g5fp?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl1z4n0gzc0yc8g3yzdz1w",
              "schemaId": "ckbeg2hzf009h0ya8324jfrmy",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl1z570gzd0yc8hgki9zki",
                "schemaId": "ckbeg2i08009v0ya8b4tj6ybl",
                "title": "Bandit",
                "value": "bandit"
              }
            }
          ]
        }
      ],
      "classifications": []
    },
    "Created By": "admin@datascone.com",
    "Project Name": "CROD",
    "Created At": "2020-06-14T21:30:52.000Z",
    "Updated At": "2020-06-14T21:30:52.000Z",
    "Seconds to Label": 173.142,
    "External ID": "image_20.png",
    "Agreement": -1,
    "Benchmark Agreement": -1,
    "Benchmark ID": null,
    "Dataset Name": "Clash-Royale-Gameplay",
    "Reviews": [],
    "View Label": "https://editor.labelbox.com?project=ckbebgxqx1ttt0766b2xxz4a2&label=ckbfl24vlxj4g0a42asrmvdwa"
  },
  {
    "ID": "ckbfl3wk2c7yu0796t754jw4e",
    "DataRow ID": "ckbed57bmbfpl0am2bf990s0i",
    "Labeled Data": "https://storage.labelbox.com/ckbeba3s31kzz0836drpx2fe8%2F8f7488d5-4151-3aa2-0d35-cbe56451606a-image_52.png?Expires=1593379948019&KeyName=labelbox-assets-key-1&Signature=08oxSs1rdj9fM_m8KGTpHj-Ncpg",
    "Label": {
      "objects": [
        {
          "featureId": "ckbfl2f9d070a0yar3eu17qto",
          "schemaId": "ckbeg2hyy009d0ya8chym8ero",
          "title": "Passive Buildings",
          "value": "passive_buildings",
          "color": "#FFFF00",
          "bbox": {
            "top": 900,
            "left": 95,
            "height": 103,
            "width": 108
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl2f9d070a0yar3eu17qto?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl2iim0gzn0yc8eaxa87aq",
              "schemaId": "ckbeg2hzi009j0ya824p6bkd7",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl2ij70gzo0yc8e5mq49se",
                "schemaId": "ckbeg2i0900a10ya8bwv2fsde",
                "title": "Elixir Collector",
                "value": "elixir_collector"
              }
            }
          ]
        },
        {
          "featureId": "ckbfl2sl00gzr0yc8dotb5ild",
          "schemaId": "ckbeg2hyy009f0ya8fdtbgk4m",
          "title": "Damaging Spells",
          "value": "damaging_spells",
          "color": "#D4FF00",
          "bbox": {
            "top": 391,
            "left": 109,
            "height": 317,
            "width": 403
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl2sl00gzr0yc8dotb5ild?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl30kd070m0yar1bwxgmm5",
              "schemaId": "ckbeg2hzn009n0ya8bo99eje2",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl30kv070n0yar1v4p7sr3",
                "schemaId": "ckbeg2i0o00av0ya87n5d3tl5",
                "title": "Freeze",
                "value": "freeze"
              }
            }
          ]
        },
        {
          "featureId": "ckbfl37cy0swr0yaoczrvcjlb",
          "schemaId": "ckbeg2hyx00990ya8a9gc465j",
          "title": "Troops",
          "value": "troops",
          "color": "#FF0000",
          "bbox": {
            "top": 380,
            "left": 137,
            "height": 93,
            "width": 85
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl37cy0swr0yaoczrvcjlb?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl3rbf0sww0yao7oflf1ku",
              "schemaId": "ckbeg2hzf009h0ya8324jfrmy",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl3rc30swx0yao9zqfhbx9",
                "schemaId": "ckbeg2i07009r0ya877wad5h0",
                "title": "Baby Dragon",
                "value": "baby_dragon"
              }
            }
          ]
        }
      ],
      "classifications": []
    },
    "Created By": "admin@datascone.com",
    "Project Name": "CROD",
    "Created At": "2020-06-14T21:32:14.000Z",
    "Updated At": "2020-06-14T21:32:14.000Z",
    "Seconds to Label": 82.172,
    "External ID": "image_52.png",
    "Agreement": -1,
    "Benchmark Agreement": -1,
    "Benchmark ID": null,
    "Dataset Name": "Clash-Royale-Gameplay",
    "Reviews": [],
    "View Label": "https://editor.labelbox.com?project=ckbebgxqx1ttt0766b2xxz4a2&label=ckbfl3wk2c7yu0796t754jw4e"
  }
]
```

We can convert to darknet format with a simple python snippet: 

```python
def write(filename, *row):
    with open(filename, "w") as file:
        file.write("\n".join(row) + "\n")

for img in json.loads(content):
    external_id = img["External ID"]
    print(f"Darknet annotations for: {external_id}")
    bulk = []
    for obj in img['Label']['objects']:
        bbox = " ".join(str(bbox_val) for bbox_val in obj['bbox'].values())
        bulk.append(f"{obj['value']}:{obj['classifications'][-1]['answer']['value']} " + bbox)
    write(external_id.replace(".png", ".txt"), *bulk)

```

Output:

```text
passive_buildings:elixir_collector 900 95 103 108
damaging_spells:freeze 391 109 317 403
troops:baby_dragon 380 137 93 85
```
The 150 images have the following class balances:

![](https://i.imgur.com/7WmHfta.png)


---
### Data augmentation
After the exhausting labeling of the 150 images, these were uploaded to Roboflow to continue with the data augmentation step. In this case, the images were flipped 180°, mirrored, and a rotated 20°. With this augmentation, a total of 450 images were generated.

![](https://i.imgur.com/PvnGx7E.png)


According to Roboflow own words:

```text
crod-test-1 - v1 2020-06-18 18:34
==============================

This dataset was exported via roboflow.ai on June 18, 2020 at 11:34 PM GMT

It includes 450 images.
Cards are annotated in YOLO v5 PyTorch format.

The following pre-processing was applied to each image:
* Auto-orientation of pixel data (with EXIF-orientation stripping)
* Resize to 416x416 (Stretch)

The following augmentation was applied to create 3 versions of each source image:
* 50% probability of horizontal flip
* 50% probability of vertical flip
* Random rotation of between -20 and +20 degrees
```
However, we identified that the amount of total distinct objects instead of being the total number of cards was only 4 which were defined by the groups of categories: Troops, Defensive Buildings, Passive Buildings, and  Spells.

A script in python was written to modify this in order to identify each card as a distinct category. 

The new file had similar annotations:

``` text
crod-test-2 - v1 2020-06-18 19:37
==============================

This dataset was exported via roboflow.ai on June 19, 2020 at 12:37 AM GMT

It includes 450 images.
Cards are annotated in YOLO v5 PyTorch format.

The following pre-processing was applied to each image:
* Auto-orientation of pixel data (with EXIF-orientation stripping)
* Resize to 416x416 (Fit (black edges))

The following augmentation was applied to create 3 versions of each source image:
* 50% probability of horizontal flip
* 50% probability of vertical flip
* Random rotation of between -20 and +20 degrees
```

### Compatibliy between LabelBox and RoboFlow

Assuming you got your JSON file on a `data` variable:

```python
import json

with open("path/to/file.json", "r") as file:
    data = json.loads(file.read())
```

You can combine the objects and categories with the following: 

```python
for d in data:
    objs = []
    if d['Label'].get('objects') is None:
        continue
    for obj in d['Label'].get('objects', [{}]):
        if not obj:
            continue
        original_value = obj["value"]
        nested_value = obj['classifications'][0]["answer"]["value"]
        value = f"{original_value}:{nested_value}"
        new_obj = {**obj, "title": value, "value": value}
        objs.append(new_obj)
    d['Label']['objects'] = objs
```

```python
with open("classification.json", "w") as file:
    file.write(json.dumps(data))
```


## Results

