---
# System prepended metadata

title: Competition

---

# Competition

## 12/1
데이터셋 접근 권한 이슈

## 12/2
데이터셋 다운로드

내일 계획
1. Instance Segmentation 모델 탐색 (SOTA 위주로 + 깃 클론해서 에러 해결까지)
2. 선정된 SOTA에 contest dataset format 변환해서 돌려보기
3. 성능 비교해서 baseline 선정
4. Google colab PRO 결제하기


## 12/3
### 모델 선정 & 환경 구축 테스트

Dexter:
1. PANet <- 개 같은 거 갖다버렸음
2. Detectron2 프레임워크 세팅 (mask-rcnn 등 [다양한 모델](https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md) 실험 가능)
    - 컴페티션 커스텀 데이터 dataset (loader) 구축 완료

Mirae:
1. Mask r cnn <- 고통 그 자체


### Code: custom dataset
```python
    from detectron2.structures import BoxMode

    classes = ['pet', 'ps', 'pp', 'pe']

    def get_contest_dicts(data_dir):
        image_dir = os.path.join(data_dir, 'image')
        label_dir = os.path.join(data_dir, 'annotation')

        dataset_dicts = []
        for obj_cls in os.listdir(label_dir):
            for i, file_name in enumerate(os.listdir(os.path.join(label_dir, obj_cls))):
                json_file = os.path.join(os.path.join(label_dir, obj_cls), file_name)

                if i < 3:
                    with open(json_file) as f:
                        label = json.load(f)
                    record = {}
                    record['file_name'] = os.path.join(image_dir, obj_cls, label['images'][0]['file_name'])
                    for value in ['height', 'width']:
                        record[value] = label['images'][0][value]
                    record['image_id'] = label['images'][0]['id']

                    objs = []
                    for instance in label['annotations']:
                        obj = {
                            "bbox": instance['bbox'],
                            "bbox_mode": BoxMode.XYWH_ABS,
                            "segmentation": instance['segmentation'],
                            "category_id": instance['category_id'] - 1,
                        }
                        objs.append(obj)

                    record['annotations'] = objs
                    dataset_dicts.append(record)

        return dataset_dicts


    for d in ["train", "test"]:
        DatasetCatalog.register("contest_" + d, lambda d=d: get_contest_dicts("contest_dataset/" + d))
        MetadataCatalog.get("contest_" + d).set(thing_classes=classes)
    contest_metadata = MetadataCatalog.get("contest_train")
```





### 내일 생각해볼 것:

- train / val / test split
- model selection

### 12/4


## 12/5




- 평가기준: COCO API에서 클래스 별 seg AP 0.50

* COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml
    * iter: 300 (default)

|   AP   |  AP50  |  AP75  |  APs  |   APm   |  APl   |
|:------:|:------:|:------:|:-----:|:-------:|:------:|
| 22.252 | 23.833 | 23.833 |  nan  | 100.000 | 20.955 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 37.086 | ps         | 51.921 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |


------------------------------------------------------
* COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml

|   AP   |  AP50  |  AP75  |  APs  |  APm  |  APl   |
|:------:|:------:|:------:|:-----:|:-----:|:------:|
| 26.815 | 27.640 | 27.640 |  nan  |  nan  | 26.815 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 40.924 | ps         | 66.337 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |

* COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml
    * iter: 3000




* LVISv0.5-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_1x.yaml

|   AP   |  AP50  |  AP75  |  APs  |   APm   |  APl   |
|:------:|:------:|:------:|:-----:|:-------:|:------:|
| 21.324 | 22.277 | 22.277 |  nan  | 100.000 | 20.041 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 32.574 | ps         | 52.723 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |


* LVISv0.5-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_1x.yaml

    * iter: 3000
    * batchsize: 256

|   AP   |  AP50  |  AP75  |  APs  |   APm   |  APl   |
|:------:|:------:|:------:|:-----:|:-------:|:------:|
| 35.594 | 35.594 | 35.594 |  nan  | 100.000 | 35.594 |

| category   | AP     | category   | AP      | category   | AP    |
|:-----------|:-------|:-----------|:--------|:-----------|:------|
| pet        | 42.376 | ps         | 100.000 | pp         | 0.000 |
| pe         | 0.000  |            |         |            |       |

---------------------------------------------------------

여기까지는 12개 데이터만 가지고 학습 후 12개 데이터만 가지고 평가한 결과.

---

#### Full dataset result
* COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml
    * iter: 3000
    * batchsize: 512
    * lr_step: 1000, 2000

|   AP   |  AP50  |  AP75  |  APs  |  APm   |  APl   |
|:------:|:------:|:------:|:-----:|:------:|:------:|
| 13.965 | 15.201 | 14.918 | 0.000 | 17.624 | 14.146 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 26.597 | ps         | 29.265 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |


### 앞으로 할 것
1. class별 AP 0 issue (bg까지 포함 class 5개로?)
1. Train / Val split 후 train 시 validation 포함 & 자동화
1. backbone / neck / head 공부 & 조사하고 커스텀
1. Data augmentation
1. LR, LRstep, Optimizer, loss 등 실험


## 12/6

- [PET, PE, PS, PP 의미](https://to-heaven.tistory.com/762)
    - PET: 폴리에틸렌 테레프탈레이트 (패트병)
    - PP: 폴리프로필렌 (요거트 용기, 지퍼락)
    - PS: 폴리스틸렌 (플라스틱 숟가락, 볼펜심)
    - PE: 폴리에틸렌

- [Base RCNN](https://github.com/facebookresearch/detectron2/blob/main/configs/Base-RCNN-FPN.yaml)

- [Detectron2 config](https://detectron2.readthedocs.io/en/latest/modules/config.html)


#### 결과
1. COCO/mask_rcnn_R_101_FPN_3x
- Params
    - batch size: 8
    - LR: 0.002
    - LR Step: 3000, 6000, 9000
    - RPN proposal size: 512
    - freeze: 0
- Iter: 1000
        
|   AP   |  AP50  |  AP75  |  APs  |  APm   |  APl   |
|:------:|:------:|:------:|:-----:|:------:|:------:|
| 14.000 | 14.284 | 14.284 | 0.000 | 94.374 | 13.751 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 25.033 | ps         | 30.966 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |

- iter: 2000

|   AP   |  AP50  |  AP75  |  APs  |  APm  |  APl   |
|:------:|:------:|:------:|:-----:|:-----:|:------:|
| 13.606 | 13.814 | 13.814 | 0.000 | 7.228 | 13.615 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 23.259 | ps         | 31.166 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |

- iter: 3000

|   AP   |  AP50  |  AP75  |  APs  |  APm   |  APl   |
|:------:|:------:|:------:|:-----:|:------:|:------:|
| 13.605 | 13.764 | 13.764 | 0.000 | 85.545 | 13.354 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 23.052 | ps         | 31.367 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |

- iter: 4000

|   AP   |  AP50  |  AP75  |  APs  |  APm   |  APl   |
|:------:|:------:|:------:|:-----:|:------:|:------:|
| 13.151 | 13.305 | 13.305 | 0.000 | 56.634 | 13.110 |

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 21.381 | ps         | 31.224 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |


2. COCO/mask_rcnn_R_101_FPN_3x
- Params
    - batch size: 8
    - LR: 0.001
    - LR Step: 2000, 4000
    - RPN proposal size: 512
    - freeze: 0
    - image size: 1024
- Iter: 500


### 앞으로 할 것

1. Data augmentation
1. backbone / neck / head 공부 & 조사하고 커스텀
1. LR, LRstep, Optimizer, loss 등 실험


## 12/8

## 12/9

- Trainer class customize
- Train, val, test split
- Validation while training
- Augmentation experiment
- Error fixed by deleting output folder (cache), but still some classes becomes 0 mAP later anyway

### 결과
1. COCO/mask_rcnn_R_101_DC5_3x.yaml
!!! contest_val로 테스트 진행!!!
- Params
    - batch size: 4
    - LR: 0.0001
    - LR Step: 2000, 4000
    - RPN proposal size: 512
    - freeze: 0
    - ITER: 5000

- Iter: 1000

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 12.141 | ps         | 13.549 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |

- Iter: 3000

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 14.206 | ps         | 14.187 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |


## 12/10
### 결과
1. COCO/mask_rcnn_R_50_DC5_3x.yaml
- Aug
    - Resize
    - RandomFlip
    - RandomBrightness
    - RandomContrast
    - RandomSaturation
    - RandomCrop
    
- Params
    - batch size: 4
    - LR: 0.001
    - LR Step: 2000, 4000
    - RPN proposal size: 512
    - freeze: 0
    - ITER: 5000

- Iter: 1000

| category   | AP     | category   | AP     | category   | AP    |
|:-----------|:-------|:-----------|:-------|:-----------|:------|
| pet        | 10.643 | ps         | 14.109 | pp         | 0.000 |
| pe         | 0.000  |            |        |            |       |

- Iter: 3000

| category   | AP    | category   | AP     | category   | AP    |
|:-----------|:------|:-----------|:-------|:-----------|:------|
| pet        | 9.061 | ps         | 14.378 | pp         | 0.000 |
| pe         | 0.000 |            |        |            |       |


- Iter: 5000

| category   | AP    | category   | AP     | category   | AP    |
|:-----------|:------|:-----------|:-------|:-----------|:------|
| pet        | 8.819 | ps         | 14.377 | pp         | 0.000 |
| pe         | 0.000 |            |        |            |       |
