---
# System prepended metadata

title: 書くこと

---


# 書くこと
1. アンサンブルの話
　マルチタスクラーニング、ターゲットごとに最適化
2. 画像前処理
　circle crop, アスペクト比　実際の画像を用いるかも？
　カメラ間でアスペクト比や網膜の映る範囲の違いを修正　→　精度向上
　使われたカメラによって眼底画像の　あれこれ
3. 画像拡張
　使った4手法を説明
　お気持ちも書こう
  Shift　→　位置による依存性を減らす
  Brightness　→　光の当たり方によらず
  Hue →　色合いの違いによらず
  
  総じてロバストなモデルのために
 
4. モデル
  EfficientNet & NFNet
　FocalLoss　→　不均衡データでもバランス良く学習
　SAM Optimizer　→　なだらかな局所解を見つけてロバスト
　CAと学習率　→　20 epochsごとに0.001と0を動く　　５５eopchs?
　Stratified KFold
5. アンサンブル
　ラベルごとに重みをランダムサーチ
6. 環境
　PyTorch　RTX 3090 & V100 x 2 & RTX 2080 :muscle: 

:scream_cat: 

# chizu & arai & okada


# 1

We propose a standard ensemble framework to combine predictions of several Convolutional Neural Networks. To make effective ensemble, we made models differ each other by choosing models with divirging complexity and trained them on different image resolution. For each of 29 targets, we used different blending weights adjusted to maximize scores on local cross-validation.

# 2

The images in the given data set were taken by three different cameras, each with different aspect ratios and retinal reflections. Therefore, we trimmed the background using Circle Crop and then aligned the aspect ratio. The idea and implementation of Circle Crop is based on [the Kaggle Notebook](https://www.kaggle.com/ratthachat/aptos-eye-preprocessing-in-diabetic-retinopathy).
Figure 1 shows an example of fundus images taken by three different cameras and the pre-processed images applied to them.

# 3

Because of data scarcity especially for rare classes, Data Augmentation(DA) plays an important role in this competition. We applied four different DAs to the images. 


To make models robust to the change of position, scale, and rotation, we used RandomResizedCrop and ShiftScaleRotate. And, to make invariant models to the change of hue and brightness, we used RandomBrightnessContrast and HueStaturationValue.


# 4

Several models were chosen from EfficientNet families and NFNet families and trained with several image resolution.
FocalLoss was employed to train models. FocalLoss dynamically weight the contribution of each samples on the loss function based on prediction difficulty, thus result in better performance on the prediction to the classes with less samples. 
We used Adam wrapped in SAM optimizer. SAM simultaneously minimizes loss value and loss sharpness, so our models became more robust.
The initial learning rate was set to 0.001 and changed using a learning rate scheduling techinique called Cosine Annealing. Cosine Annealing changes learning rate $\eta_t$ to follow the value of the following function

$$
    \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right) \right)
$$

We set $\eta_{min}=0.0$, $\eta_{max}=0.001$, $T_{max}=10$ and trained the models for 55 epochs.

# 5

To ensemble the predictions of each model, we used weighted blending. Blending weights are ajusted for each target. We applied random search to optimize the weights by maximizing the local cross validation score for each target.

# 6

Implementation of this work was done in PyTorch v1.7. The training was performed using a NVIDIA GeForce RTX 3090, two NVIDIA Tesla V100 with 32GB vRAM, and one NVIDIA GeForce RTX 2080. The source code for training is available at https://github.com/koukyo1994/riadd-competition and https://github.com/Chizuchizu/riadd. The source code for inference is available at https://www.kaggle.com/hidehisaarai1213/riadd-inference.
