# CVPR 2020

## Experiments

#### TODO

- [x] change loss to multi-sigmoid
- [ ] Optimize dataloader
- [x] Shuffle training and validation set to check if the low validation MIOU is due to model overfit to training rasters.
- [ ] Visualization of final MIoU
- [x] Check weight calculation function
- [ ] Regular ways to prevent overfit: Train augmentation (add randomness, avoid overfitting), weight decay
- [x] recalculate class weights: balance each class positive and negative samples
- [ ] modify BCEwithlogits use new calculated weights：loss weight 怎么改的？
- [ ] 拼接出大图，看每张图的整体和每一类的histogram：每一类真正的feature是什么？？？

#### TODO:
- [ ] exp25: 6000 more (+ scale) + augmentation. (follow up to exp22)
- [ ] exp26: mix (more training) + augmentation. (follow up to exp23)
- [ ] exp27: mix (more training) + 6000 more (+ scale) + augmentation

Generate mix dataset in NAS, and generate a new dictionary?

#### 0. Baseline model 2.25
- Using GID-5 class backbone.
- lr 1, wd 0, epoch 100
- Model path: `/media/workspace/nanqiao/agri_code/pytorch_deeplab/run/cvpr2020/gid_4ch/experiment_lr1_wd0_ep100_class6`
- Confusion matrix of validation and training set. Val miou **0.36**.

:exclamation:

#### 1. Change loss function to BCEWithSigmoid 2.27

[[646544949 15444447 2273867 242562 4413083 1884878 96608119]
[ 8324497 16759121 35792 1 90130 63017 363963]
[ 2730632 5945 2518004 103 30503 0 771380]
[ 283896 0 1450 26244 12727 0 3801]
[ 6863101 553967 10014 818 9860376 477 128919]
[ 6318979 10132 8461 0 16950 8777171 1665550]
[ 59776985 723633 835253 7015 45542 256005 147123863]]
before mean [0.75911396 0.39549879 0.27306081 0.04535643 0.44765751 0.46191698
0.47719454]
after mean **0.4085427177397207**

[[2024955001, 32976957, 10980940, 1275202, 8888387, 7866752, 132559635]
[ 247367, 109852475, 173, 0, 0, 40, 2127]
[ 349447, 586, 24330545, 824, 56, 238, 13250]
[ 24308, 4581, 469 , 7287584, 117, 50, 173]
[ 332456, 19, 8 , 0, 54769950, 730, 12668]
[ 80847, 56, 918, 80, 222, 41842021, 26481]
[ 7186276, 219861, 110312, 1433, 33235 , 68973, 498328791]]
before mean [0.90897947 0.7665682 0.67985649 0.8479041 0.85527468 0.8387291 0.78039069]
after mean 0.8111003925621956

#### 2. Change backbone to GID -> GID+Cropland (3 class) 2.29

[[651429608 12049778 1901635 97167 4852677 2039499 95041541]
[ 6927007 18351708 4 0 2533 110114 245155]
[ 2119957 36328 3333738 0 2228 2826 561490]
[ 275339 3416 1143 27667 14953 604 4996]
[ 5343963 409873 3375 0 11581198 1341 77922]
[ 5432905 42348 600 0 89 10309881 1011420]
[ 51289835 1369791 162507 1089 22430 168975 155753669]]
before mean [0.77662005 0.46403566 0.41026425 0.06488904 0.51904338 0.53920274 0.50948039]
after mean **0.46907650420171504**

<img src="CVPR 2020.assets/image-20200404114532868.png" alt="image-20200404114532868" style="zoom:80%;" />

#### 3. Shuffled training set and validation set

- val miou ~0.6
- Generalize to unseen area!


#### 4. Change loss to sum of 7 BCE loss w/o normalization

def MultiLabelBCEWithLogitsLoss(self, logit, target):
    n_class = logit.shape[1]
    loss = 0.0
    for c in range(n_class):
        criterion = nn.BCEWithLogitsLoss(pos_weight=self.weight[c], reduction='mean')
        if self.cuda:
            criterion = criterion.cuda()
        loss += criterion(logit[:,c], target[:,c])
    return loss

Using new weights.
Failed. MIoU drops to 0.


#### 5. Change loss to sum of 7 BCE loss w/o normalization and normlized by (1 + weight)

def MultiLabelBCEWithLogitsLoss(self, logit, target):
    # (b, c, h, w)
    n_class = logit.shape[1]
    loss = 0.0
    for c in range(n_class):
        criterion = nn.BCEWithLogitsLoss(pos_weight=self.weight[c], reduction='mean')
        if self.cuda:
            criterion = criterion.cuda()
        loss += criterion(logit[:,c], target[:,c]) / (self.weight[c] + 1) # (target[:,c,:,:]), (b, h, w), weight[c]
    return loss

Using new weights
run_id: `lr1e-1_wd1e-5_ep200_bs176_cuda_lossnorm`


#### 6. Change loss to sum of 7 BCE loss w/o normalization and normlized by `*(1+weight)/weight`

def MultiLabelBCEWithLogitsLoss(self, logit, target):
    # (b, c, h, w)
    n_class = logit.shape[1]
    loss = 0.0
    for c in range(n_class):
        criterion = nn.BCEWithLogitsLoss(pos_weight=self.weight[c], reduction='mean')
        if self.cuda:
            criterion = criterion.cuda()
        loss += criterion(logit[:,c], target[:,c]) * (self.weight[c] + 1) / self.weight[c]
    return loss

Using new weights.
Failed. Very similar to Exp4. Since when the weight is too large, `(self.weight[c] + 1) / self.weight[c]` is very close to `weight[c]`


#### 7. Train a model specific for planter_skip

- What loss? cross entropy with `ignored_index`
- What's the label looks like? 1 v.s. the rest. Ignore index out of `mask` and `boundary`.
- Failed. MIOU ~ 0.5.

#### 8. Use cross entropy loss. Ignore multi-label problem

- Weight: [ 1.56581043, 7.33404178, 50.49834979, 50.49834979, 50.49834979, 50.49834979, 50.49834979]


MIOU doesn't increase. Due to bad weight.

#### 9. Use BCE loss, aiming to reproduce exp2, batch size 100 (4GPU)

- Weight: [ 1.56581043, 7.33404178, 50.49834979, 50.49834979, 50.49834979, 50.49834979, 50.49834979]
- Failed. Due to bad weight

#### 10. Follow up to exp9, increase batch size to 176, aiming to reproduce exp2

- Weight: [ 1.56581043, 7.33404178, 50.49834979, 50.49834979, 50.49834979, 50.49834979, 50.49834979]
- Failed. Due to bad weight


#### 11. Follow up to exp10, change weight-decay to 0, aiming to reproduce exp2

- Weight: [ 1.56581043, 7.33404178, 50.49834979, 50.49834979, 50.49834979, 50.49834979, 50.49834979]
- Failed. Due to bad weight

#### 12. Follow up to exp11, use a new weight, aiming to reproduce exp2.

- Use new weight: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748, 5.7252]
- `bs=176, lr=1, wd=0`
- ```python
def BCEWithLogitsLoss(self, logit, target):
"""
Add layer weight
:param logit:
:param target:
:return:
"""
criterion = nn.BCEWithLogitsLoss(pos_weight=self.weight, reduction='mean') # (c, h, w)
if self.cuda:
criterion = criterion.cuda()
loss = criterion(logit, target)
return loss

![](https://i.imgur.com/EMWpPWQ.png)

- Achieved performance close to exp2.

#### 13. Change loss to multi-BCE loss

- Use new weight: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748,  5.7252]

- `bs=176, lr=1, wd=0`

- ```python
  def MultiLabelBCEWithLogitsLoss(self, logit, target):
      # (b, c, h, w)
      n_class = logit.shape[1]
      loss = 0.0
      for c in range(n_class):
          criterion = nn.BCEWithLogitsLoss(pos_weight=self.weight[c], reduction='mean')
          if self.cuda:
              criterion = criterion.cuda()
          loss += criterion(logit[:,c], target[:,c]) * (self.weight[c] + 1) / self.weight[c] # (target[:,c,:,:]), (b, h, w), weight[c]
      return loss


- Start to predict all "background" class after epoch 55.

#### 14. Follow up to exp13, no normalization to loss

- Use new weight: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748, 5.7252]
- `bs=176, lr=1, wd=0`
- ```python
def MultiLabelBCEWithLogitsLoss(self, logit, target):
# (b, c, h, w)
n_class = logit.shape[1]
loss = 0.0
for c in range(n_class):
criterion = nn.BCEWithLogitsLoss(pos_weight=self.weight[c], reduction='mean')
if self.cuda:
criterion = criterion.cuda()
loss += criterion(logit[:,c], target[:,c]) # (target[:,c,:,:]), (b, h, w), weight[c]
return loss

![](https://i.imgur.com/NZZXUoa.png)

- Best MIoU ~0.46. 
- Compare to exp12, two models have similar behavior although trained with different loss type. 

#### 15. Use BCE loss and "original weight", try to reproduce exp2

- Use weight  [ 1.4262, 50.1726, 50.0267, 50.4068, 50.2656, 50.0574, 43.4591]
- `bs=176, lr=1, wd=0`: same as exp2
- Use BCE loss

![](https://i.imgur.com/aH2G5LV.png)

- Model starts to predict all "weed cluster" after 50 epoch. Due to the weight of "weed cluster" is large (43.4591). 

#### 16. Use CE loss and weight D
- Use new weight: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748,  5.7252]
- CE loss. `bs=176, lr=1, wd=0`

![](https://i.imgur.com/f9ukL9j.png)

- Miou is low.

#### 17. Crop 1000 more patches from merged images.
- Recalculate weight E: [ 1.7473, 17.6824, 34.9618, 41.2901, 25.1894, 29.9803, 5.9957]
- MBCE loss, fine tune from exp14
![](https://i.imgur.com/EA7Y9oq.png)
- MIOU goes down. Seems to overfit. Might due to newly cropped patches are very close to samples in training set.

#### 18. Follow up to exp17, crop another 5000 patches. (6000 new patches totally)
- Recalculate weight F: [ 1.7398, 15.7913, 32.0508, 31.7640, 21.8078, 28.0981, 7.1074]
- MBCE loss, fine tune from exp14
![](https://i.imgur.com/M46hC0Q.png)
- Similar to exp17, overfit. 


#### 19. Augmentation on the fly.
- Use original training set. Use weight D
- Add augmentation on the fly (random flip and rotation).
- Fine tune from exp14 for 50 epoch
![](https://i.imgur.com/3ygKnb3.png)
- MIOU increased to 0.476. 
- Submitted once using this checkpoint, result: 0.462 (dropped from 0.468 using exp2 checkpoint)

#### 20. Redo: train a model specific for planter-skip
- Select 270 planter-skip samples from original dataset.
- Recalculate weight: [1.xxxx, 8.xxxx]
- Augment on the fly.
- Fine tune from exp14
![](https://i.imgur.com/R87jMPy.png)
- Bad result, miou < 0.5. Unable to recognise planter-skip
- Run test script:
    - CM:[[10297971  133667]
        [  218487   109631]]
    - before mean [0.86879746 0.06585267]
    - after mean 0.46732506661177825
- Validation set prediction: sftp://209.172.89.185/media/workspace/nanqiao/exp_res/cvpr2020/vis_exp20

#### 21. Follow up to exp18 and exp19, 6000 more samples + augmentation
- Use weight F
- Fine tune from exp14 for 50 epoch
![](https://i.imgur.com/FX3HBx3.png)
- MIOU increased to 0.478

#### 22. Follow up to exp21, train from scratch
- 6000 more samples + augmentation, train from scratch
- Use weight F
![](https://i.imgur.com/Jo77zXt.png)
- Val MIOU 0.489
- Run test script on validation set
    [[657102154   9801041   1982820    346348   8520737   2168865  87489940]
     [  6844762  17809022       240         0    229805     75155    677537]
     [  1834785      1578   3798386         0     63065      1131    357622]
     [   262853      3772      1494     49030         0      2792      8177]
     [  4523919     86587      3283      1206  12716178       921     85578]
     [  4277167     53403       173         0       532  11374329   1091639]
     [ 53532401    756199    147121     28612     75444    370434 153858085]]
    before mean [0.7834884  0.49007877 0.46368726 **0.0696168** 0.48337153 0.58580614 0.5154741 ]
    after mean 0.4845032858388702
- Run test script on training set
[[2128362341   11479974    5827226     419075    4390920    6745983 62277355]
 [   3319488  106766666        510          0          0       8067 7451]
 [    630932        523   24041564        205          5       1737 19980]
 [     39611          0        291    7277173          0         99 108]
 [    484615        235         62          0   54623984        988 5947]
 [    118905          0         78         11         45   41824987 6599]
 [  30328316     131906      99689       1423      39395     219828 475128324]]
before mean [0.94408223 0.87718707 0.78508701 **0.94044673** 0.91733793 0.85483899 0.8361015 ]
after mean 0.8792973506996555
- Run test script on validation set using HALF precision
[[673710595   8898418   1524392    236862   4094547   2401351  76545740]
 [  7678721  17223363       104      1211    143363     92864    496895]
 [  2156773      1699   3570140         0      2211       799    324945]
 [   279057       450      1800     32721      7481      2757      3852]
 [  5505817    109100      1168      4004  11741739      1679     54165]
 [  4700454     59136       280         0         0  11284593    752780]
 [ 62574849   1128185    200474     18200     27360    520708 144298520]]
before mean [0.792314   0.48064963 0.45860483 0.0556106  0.5412777  0.5694285
 0.50287574]
after mean 0.48582300057220273
- Submit result: 
![](https://i.imgur.com/z6vCM6N.png)

#### 23. Mix train, val set, train from scratch
- Use weight: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748,  5.7252]
- Use mbce loss, train from scratch
![](https://i.imgur.com/XKZOUY0.png)
- MIOU 0.4812
- This one might be a **fake mix**

#### 24. Original dataset, add 6000 samples (including scale).
- Use weight [ 1.7398, 15.7913, 32.0508, 31.7640, 21.8078, 28.0981, 7.1074]
- Use mbce loss, train from scratch
- Add apex (mix precision training) in this version
    - if do not use apex: float32, 40 min/epoch
    - apex opt_level = 'O1': 14 min/epoch
    - apex opt_level = 'O3': 13 min/epoch
- apex level = 'O1'
![](https://i.imgur.com/o9VnqaQ.png)
- The batch size in this exp turns out to be 10. **BUG!**


#### 25. Redo exp24 with batch-size = 176
- The goal of this experiment is to make sure the "scale" augmentation works well.
![](https://i.imgur.com/cpsK3jT.png)
- MIOU: 0.468. Lower than exp22

#### 26. shuffle_1: Mix dataset (14900 training)
- weight: [1.7569, 19.4266, 37,0463, 46.1138, 26.7562, 29.9496, 5.5902]
- wd: 1e-5


#### 27. shuffle_2: Mix dataset (14900 training) + 7000 new samples
- weight: [ 1.7405, 16.9784, 31.6730, 31.6574, 22.0800, 28.0861,  6.8873]
- wd: 1e-4, 

![](https://i.imgur.com/Yx83dxr.png)

- validation set:
    - sftp://209.172.89.185/media/workspace/nanqiao/exp_res/cvpr2020/vis_exp27_val
 [[385864983   4933113   1766164    182934   1388343   1648863  26064349]
 [  1821723  17737871      4732       133         0     14080     50875]
 [   557580        65   4070867         0       586       222     59919]
 [   166148         2       406    799781       619         0      3701]
 [   705343      3175         0         0   9366146      3189     18753]
 [   521026       121       232        17       112   7354038    124020]
 [ 12896752     56606     55451       333      4491    109575  85541491]]
before mean [0.879931   0.72039289 0.62472791 0.69300669 0.81510261 0.75229316 0.68440685]
after mean 0.738551585480104

- training set
[[3380123884   59351858   19269072    6653707   25464395   17908028
   161229605]
 [   6047879  196599699       2705       1182          0       3666
       15442]
 [   3700662       3616   56300443       7390        335       4756
      159594]
 [    565714         10        913   58793698       1135        295
        4837]
 [   1700830       1754          0          0  126649887        283
       28240]
 [    985614       3412       5068       3049       9849   81822161
       30692]
 [  61000885     573256     281461      34264     424325    1204878
   598982840]]
before mean [0.90281035 0.74865326 0.70608549 0.88992107 0.8209038  0.80232159 0.72694711]
after mean 0.7996632390570124

- test set predictions
    - sftp://209.172.89.185/media/workspace/nanqiao/exp_res/cvpr2020/vis_exp27_test
![](https://i.imgur.com/7aJ8ts9.png)

- validation set v2

#### 28. Follow up to exp27, fine tune using lovasz loss
![](https://i.imgur.com/E96MFoY.png)
- MIOU: 0.759

#### 29. Add 7000 more samples to shuffled_2, total 28307 in training set
- New weight: [ 1.73516535 19.25849021 31.26115109 28.18704187 18.01456648 27.13094251 7.43372477]
- lr finder result seems not good

#### 30. Combine training and validation set, add 8000 samples(rotation + flip)
- weight [ 1.7422943  22.51612081 34.54451569 34.65533898 18.05498276 28.27384239 6.44964741]
- 2524 samples




| Exp. | loss type | weight | MIoU |
| -------- | -------- | -------- | -------- |
| 0 (GID backbone)    | ce      | A     |0.36|
| 1 (GID backbone)    | bce      | A     |0.41|
| 2     | bce     | A     |0.47|
| 3 (shuffled dataset)    | bce      | A     |0.60|
| =gap=     |      |      
| 4     | mbce     | C     |0.25 -> 0|
| 5     | mbce     | C /(w + 1)    |0.35|
| 6     | mbce     | C \*(w + 1)/w     |0.22 -> 0|
| 7 (only background and planter skip)    | ce     | B     |~0.5 (failed, only two class)|
| 8     | ce     | B     |0.16|
| 9 (bs100, wd1e-5)    | bce     | B     |somenumber -> 0|
| 10 (bs176, wd1e-5)    | bce     | B     |somenumber -> 0|
| 11 (bs176, wd0)    | bce     | B     |somenumber -> 0|
| 12 (bs176, wd0)    | bce     | D     |0.45|
| 13     | mbce     | D \*(w + 1)/w   |0.43 -> 0|
| 14     | mbce     | D   |0.46|
| 15     | bce     | A   |0.34 -> 0|
| 16     | ce     | D   |0.26|
| 17 (finetune from 14)(1000+ training)    | mbce     | E   |<0.46|
| 18 (finetune from 14)(6000+ training)    | mbce     | F   |<0.46|
| 19 (finetune from 14)(random flip/rotation)    | mbce     | D   |0.476|
| 20 (finetune from 14)(only 270 planter-skip samples)    | ce     | [1, 8]   |<0.5|
| 21 (finetune from 14)(6000+ training + augmentation)    | mbce     | F   |0.478|
| 22 (training from scratch)(6000+ training + augmentation)    | mbce     | F   |0.489|
| 23 (has bug, ignore)    | mbce     | G   |...|
| 24 (has bug, ignore)    | mbce     | G   |...|
| 25 (training from scratch)(6000+ (add scale) training + augmentation)    | mbce     | H   |0.468|
| 26 (shuffle_1, mix dataset)    | mbce     |    |~0.7|
| 27 (shuffle_2, mix dataset)(7000+ (add scale) training + augmentation)    | mbce     |    |0.738|
| 28 (fine tune from 27)    | lovasz     |    |0.759|
| 29 (add another 7000 to shuffled_2)    | mbce     | G   |...|

- Some conclusions:
    - exp2(submission 0.47)/exp14 v.s. exp22 (submission 0.49): Generating more data (rotation + flip) and augmentation helps. 
    - exp22 (submission 0.49) v.s. exp27 (submission 0.54/0.55): More data from validation set helps. After mixing training and validation set , and putting more samples in training set, submission result increases.

- Questions need to find out:
    - Does scale benefit our model? Compare 22 and 25, validation miou drops after adding scale. Submit exp25 to the leader board! **(exp25: 0.4919, exp22: 0.4909)** Then we can answer another question: Does a model perform similarly in original validation and test set. (previously, it does: exp14 v.s. exp22) 
    - Does lovasz loss helps? Compare exp27 and exp28, "validation set" is better (0.738 -> 0.759) while submission is worse (0.540 -> 0.524). This also tells us the current "validation set" and test set may not act the same.
    - Can we add more date and get more benefits? 
        - If we run exp in mix dataset, the only way to know if it helps is submission
        - We can also run exp following up to exp22, then we may know through validation set.



- weight
    * A: [ 1.4262, 50.1726, 50.0267, 50.4068, 50.2656, 50.0574, 43.4591] This is the one I copied from Section Thoughts. 
    * B: [ 1.56581043, 7.33404178, 50.49834979, 50.49834979, 50.49834979, 50.49834979, 50.49834979] This is the one copied from console.
    * C: [ 0.5237, 29.7162, 135.9478, 461.1825, 60.3602, 79.6167, 5.6843] This is a weight calculated by Yi. 
    * D: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748, 5.7252] This is the latest weight calculated by Nan.
    * E: [ 1.7473, 17.6824, 34.9618, 41.2901, 25.1894, 29.9803,  5.9957]. Calculated on 4/9. Add 1000 samples to training set. 
    * F: [ 1.7398, 15.7913, 32.0508, 31.7640, 21.8078, 28.0981,  7.1074]. Calculated on 4/9. Add another 5000 samples to training set. 
    * G: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748, 5.7252]. New weight for mixed dataset.
    * H: [ 1.73751758 16.2182834  32.1548142  31.6837071  21.59519342 28.18540157 7.10395381]. Weight for training set + 6000 samples (including scale)
    * I: [ 1.73516535 19.25849021 31.26115109 28.18704187 18.01456648 27.13094251 7.43372477]. Weight for mixed training set + 14000 samples
    * J: [ 1.7561, 19.3694, 36.9336, 46.1758, 26.7731, 30.0547,  5.5229]
- loss type:
    - ce: cross entropy
    - bce: binary cross entropy (weight is applied to differnent layer)
    - mbce: multi 
- Comparison
    - 11, 12: weight B is bad
    - 12, 13: loss mbce  *(w + 1)/w is not working.


## Analysis

#### 1. Visualize 3-planter_skip in training and validation set

- sftp://209.172.89.185/media/workspace/nanqiao/exp_res/cvpr2020/vis_planter_skip
- Some confusing annotations

#### 2. Visualize normalized logits

- Logits for each image are normalized as a CxHxW block, so that the value is comparable in the final argmax operation.
- Shuffled validation set (better result): sftp://209.172.89.185/media/workspace/nanqiao/exp_res/cvpr2020/vis_normalized_logits_shuffle_val
- Original validation set (worse result): sftp://209.172.89.185/media/workspace/nanqiao/exp_res/cvpr2020/vis_normalized_logits
  - label for 7 classes, logits for 7 classes, rgb image, prediction after argmax.


#### 3. A multiplier on "0-background" logits
| multiplier | val miou |
| ---------- | -------- |
| 1          | 0.469    |
| 0.9        | 0.466    |
| 0.8        | 0.463    |
| 0.6        | 0.457    |

#### 4. Visualization: merge image patches

结果在：/media/workspace/nanqiao/exp_res/cvpr2020/merged_rgb
其中 `\*_rgb` 是training validation test 的全部rgb图拼起来的结果；、`*_all`是从training和validation中sample出一些大图，画出了rgb、nir、mask、boundary、以及六类标注的结果

#### 5. Check label overlaps:

4400 * 512 * 512 = 1e9
1e5
1e4

- validation set: 
  (1, 6): 90381, 
  (2, 3): 3,  
  (3, 6): 1205, 
  (4, 6): 46349, 
  (5, 6): 168251
- training set
  (1, 2): 10902, (1, 6): 1749327,
  (2, 3): 684, (2, 4): 140,(2, 5): 1448, (2, 6): 981598,
  (3, 4): 22009, (3, 5): 243， (3, 6): 7216,
  (4, 5): 23105, (4, 6): 278693, 
  (5, 6): 566713, 

## Data
Label map legends:

 <img src="https://i.imgur.com/ZYw7XzB.png =100x" style="zoom:50%;" />

## Paper
- Image processing
  ![](https://i.imgur.com/AXcmQKE.png)
- Annotation:
  - During annotation, they refered RGB, Nir and **NDVI**
- Image sample generation
  - For field patterns smaller than the window size, we simply crop the region centered at the annotation
  - For field patterns larger than the window size, we employ a non-overlapping sliding window technique to cover the entirety of the annotation.
    - Note that we discard images covered by more than 90% of annotations, such that all images retain sufficient context 

loss = sigma (-y log y_hat)

y =     [0,     1,    1,    0]
y_hat = [0.1, 0.8, 0.05, 0.05]

logits: [-10, 5, 4, -2]


## Thoughts

- Imbalanced dataset

  - Our current weight [ 1.4262, 50.1726, 50.0267, 50.4068, 50.2656, 50.0574, 43.4591]
  - Calculated by

class_weight = 1 / (np.log(1.02 + (frequency / total_frequency)))

  - Paper
    - [A systematic study of the class imbalance problem in convolutional neural networks](https://arxiv.org/pdf/1710.05381.pdf)
      - Oversampling
  - Might still be a problem, see validation result in Exp2. 

- Overfit:

  - Training data augmentation
  - TTA

- check entropy, considering use pointrend: this may lead to a more refined segmentation result

## New weights

- Weights are calculated by every single class. Weights among different classes don't correlate to each other. 
- e.x., the number '0.5237' at the first row means, for class 0, positive samples should have weight '0.5237', compared to negative samples.
- class weight **([  0.5237,  29.7162, 135.9478, 461.1825,  60.3602,  79.6167,   5.6843])**
- Stored at `/NAS/datasets/Agriculture-Vision/cvpr2020_classes_weights_binary.npy`

### New weight 4/4

Calculated 4/4 using training set: [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748,  5.7252]

Previously used: 

### Generate more data
- Do 1000 times:
  - Sample a class using weight  [ 1.7500, 18.3177, 36.1873, 45.0451, 26.4434, 30.5748,  5.7252]
  - Sample a patch contain this class
  - Find this patch in the large image, shift (-50 - 50) in both x and y direction, crop the patch with the new range and put it into training set.
- Weight becomes to [ 1.7473, 17.6824, 34.9618, 41.2901, 25.1894, 29.9803,  5.9957] after merge these samples with original training set.
- Do another 5000 times:
- Weight becomes to [ 1.7398, 15.7913, 32.0508, 31.7640, 21.8078, 28.0981, 7.1074] after merge these samples with original training set.

### Model size
original 246MB
apex O1, O3: 237.9MB

### Number of new samples and miou gain.
- Old: exp 2
    > miou: 0.468151
    background: 0.768111
    cloud_shadow: 0.351518
    **double_plant: 0.324109**
    **planter_skip: 0.105950**
    standing_water: 0.651003
    waterway: 0.546880
    weed_cluster: 0.529484
- New: exp 27
    >miou: 0.549545
    background: 0.798900
    cloud_shadow: 0.386338
    **double_plant: 0.476386**
    **planter_skip: 0.261580**
    standing_water: 0.746414
    waterway: 0.620535
    weed_cluster: 0.556659
- sample prior [ 1.7500, 18.3177, **36.1873**, **45.0451**, 26.4434, 30.5748, 5.7252]

### Split