# CROD: Labeling Process

This document contains a description of the labeling process used for the proof-of-concept.

## Motivation

We are using a computer vision technique for object detection called YOLO (you only look once) that’s prominent on fast inference and accurate training. In order to develop a proof of concept for this model, we need a sample dataset to train, test, and validate.

## Methodology

To get a training dataset, we screen recorded a few ClashRoyale games from a single device (Samsung Galaxy A8+ 2018 using Android version 9). In total, we got 5 gameplay videos available on [this google drive folder](https://drive.google.com/drive/u/2/folders/1V2AEtfUliHC0gfdynMGyCepHjgs53ER1). We split apart the video into images by using a uniform time unit of 1 second. After that, we took 30 random images out, giving us a total of 150 images as training samples.

Once having our custom dataset, we used [LabelBox](https://app.labelbox.com/) for the tagging and labeling of bounding boxes. The configuration of the image labeler contains the following object:

* Troops: 73 total
* Archers
* Baby Dragon
* Balloon
* +70 more

* Defensive Buildings: 6 total
* Bomb Tower
* Cannon
* Inferno Tower
* +3 more

* Passive Buildings: 6 total
* Barbarian Hut
* Elixir Collector
* Furnace
* +3 more

* Spells: 17 total
* Arrows
* Barbarian Barrel
* Clone
* +14 more

These categories were defined by using the following source:
* Stats Royale: https://statsroyale.com/en/cards
* ClashRoyale Fandom: https://clashroyale.fandom.com/wiki/Cards

Once the labeling process is finalized, we can export the results as a json file similar to the follownig:

[
  {
    "ID": "ckbfl24vlxj4g0a42asrmvdwa",
    "DataRow ID": "ckbed57bjbfm10am244wv56z5",
    "Labeled Data": "https://storage.labelbox.com/ckbeba3s31kzz0836drpx2fe8%2F7bc64cb5-ff96-1a55-b17a-e3e8dbf35616-image_20.png?Expires=1593379948018&KeyName=labelbox-assets-key-1&Signature=FnGPV2BXd40lbXIMP2YeaPPgztE",
    "Label": {
      "objects": [
        {
          "featureId": "ckbfl1va00gz90yc83v20g5fp",
          "schemaId": "ckbeg2hyx00990ya8a9gc465j",
          "title": "Troops",
          "value": "troops",
          "color": "#FF0000",
          "bbox": {
            "top": 494,
            "left": 524,
            "height": 155,
            "width": 112
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl1va00gz90yc83v20g5fp?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl1z4n0gzc0yc8g3yzdz1w",
              "schemaId": "ckbeg2hzf009h0ya8324jfrmy",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl1z570gzd0yc8hgki9zki",
                "schemaId": "ckbeg2i08009v0ya8b4tj6ybl",
                "title": "Bandit",
                "value": "bandit"
              }
            }
          ]
        }
      ],
      "classifications": []
    },
    "Created By": "[e-mail]",
    "Project Name": "CROD",
    "Created At": "2020-06-14T21:30:52.000Z",
    "Updated At": "2020-06-14T21:30:52.000Z",
    "Seconds to Label": 173.142,
    "External ID": "image_20.png",
    "Agreement": -1,
    "Benchmark Agreement": -1,
    "Benchmark ID": null,
    "Dataset Name": "Clash-Royale-Gameplay",
    "Reviews": [],
    "View Label": "https://editor.labelbox.com?project=ckbebgxqx1ttt0766b2xxz4a2&label=ckbfl24vlxj4g0a42asrmvdwa"
  },
  {
    "ID": "ckbfl3wk2c7yu0796t754jw4e",
    "DataRow ID": "ckbed57bmbfpl0am2bf990s0i",
    "Labeled Data": "https://storage.labelbox.com/ckbeba3s31kzz0836drpx2fe8%2F8f7488d5-4151-3aa2-0d35-cbe56451606a-image_52.png?Expires=1593379948019&KeyName=labelbox-assets-key-1&Signature=08oxSs1rdj9fM_m8KGTpHj-Ncpg",
    "Label": {
      "objects": [
        {
          "featureId": "ckbfl2f9d070a0yar3eu17qto",
          "schemaId": "ckbeg2hyy009d0ya8chym8ero",
          "title": "Passive Buildings",
          "value": "passive_buildings",
          "color": "#FFFF00",
          "bbox": {
            "top": 900,
            "left": 95,
            "height": 103,
            "width": 108
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl2f9d070a0yar3eu17qto?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl2iim0gzn0yc8eaxa87aq",
              "schemaId": "ckbeg2hzi009j0ya824p6bkd7",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl2ij70gzo0yc8e5mq49se",
                "schemaId": "ckbeg2i0900a10ya8bwv2fsde",
                "title": "Elixir Collector",
                "value": "elixir_collector"
              }
            }
          ]
        },
        {
          "featureId": "ckbfl2sl00gzr0yc8dotb5ild",
          "schemaId": "ckbeg2hyy009f0ya8fdtbgk4m",
          "title": "Damaging Spells",
          "value": "damaging_spells",
          "color": "#D4FF00",
          "bbox": {
            "top": 391,
            "left": 109,
            "height": 317,
            "width": 403
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl2sl00gzr0yc8dotb5ild?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl30kd070m0yar1bwxgmm5",
              "schemaId": "ckbeg2hzn009n0ya8bo99eje2",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl30kv070n0yar1v4p7sr3",
                "schemaId": "ckbeg2i0o00av0ya87n5d3tl5",
                "title": "Freeze",
                "value": "freeze"
              }
            }
          ]
        },
        {
          "featureId": "ckbfl37cy0swr0yaoczrvcjlb",
          "schemaId": "ckbeg2hyx00990ya8a9gc465j",
          "title": "Troops",
          "value": "troops",
          "color": "#FF0000",
          "bbox": {
            "top": 380,
            "left": 137,
            "height": 93,
            "width": 85
          },
          "instanceURI": "https://api.labelbox.com/masks/feature/ckbfl37cy0swr0yaoczrvcjlb?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJja2JlYmEzc2s1bTQ2MDcyMDhuMjVwaWc3Iiwib3JnYW5pemF0aW9uSWQiOiJja2JlYmEzczMxa3p6MDgzNmRycHgyZmU4IiwiaWF0IjoxNTkyMTcwMzQ3LCJleHAiOjE1OTQ3NjIzNDd9.aMGGYEmPSpBmgUEZZiXPaO2SDZT_95m2Kc7vkIZUe8M",
          "classifications": [
            {
              "featureId": "ckbfl3rbf0sww0yao7oflf1ku",
              "schemaId": "ckbeg2hzf009h0ya8324jfrmy",
              "title": "Select card type:",
              "value": "select_card_type:",
              "answer": {
                "featureId": "ckbfl3rc30swx0yao9zqfhbx9",
                "schemaId": "ckbeg2i07009r0ya877wad5h0",
                "title": "Baby Dragon",
                "value": "baby_dragon"
              }
            }
          ]
        }
      ],
      "classifications": []
    },
    "Created By": "[e-mail]",
    "Project Name": "CROD",
    "Created At": "2020-06-14T21:32:14.000Z",
    "Updated At": "2020-06-14T21:32:14.000Z",
    "Seconds to Label": 82.172,
    "External ID": "image_52.png",
    "Agreement": -1,
    "Benchmark Agreement": -1,
    "Benchmark ID": null,
    "Dataset Name": "Clash-Royale-Gameplay",
    "Reviews": [],
    "View Label": "https://editor.labelbox.com?project=ckbebgxqx1ttt0766b2xxz4a2&label=ckbfl3wk2c7yu0796t754jw4e"
  }
]

We can convert to darknet format with a simple python snippet:

def write(filename, *row):
    with open(filename, "w") as file:
        file.write("\n".join(row) + "\n")

for img in json.loads(content):
    external_id = img["External ID"]
    print(f"Darknet annotations for: {external_id}")
    bulk = []
    for obj in img['Label']['objects']:
        bbox = " ".join(str(bbox_val) for bbox_val in obj['bbox'].values())
        bulk.append(f"{obj['value']}:{obj['classifications'][-1]['answer']['value']} " + bbox)
    write(external_id.replace(".png", ".txt"), *bulk)

Output:

passive_buildings:elixir_collector 900 95 103 108
damaging_spells:freeze 391 109 317 403
troops:baby_dragon 380 137 93 85

## Results