---
# System prepended metadata

title: Benchmark on CIFAR10 (50k+10k)

---

# Benchmark on CIFAR10 (50k+10k)

## Table of Contents
[TOC]

## Models
  - GreenAINet (porting from gv856.py)
  - [MobileNetV2](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/mobilenetv2.py#L40)
  - [ResNet18](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/resnet.py#L100)
  - [ResNet34](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/resnet.py#L103)
  - [VGG11](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/vgg.py#L14)

This benchmark adopts MobileNetV2, ResNet18, ResNet34 and VGG11 from this [github repo](https://github.com/kuangliu/pytorch-cifar).

## Environment & Configuration

  - HW/SW
      - Python3.6
      - Pytorch1.0
      - Graphic card: GeForce GTX 1080Ti on spd7
  - Training Config
      - Use no Data Augmentation
      - 50 epochs
      - Loss function: [`MultiMarginLoss(p=2)`](https://pytorch.org/docs/stable/nn.html#multimarginloss)
      - Optimize method: [`SGD`](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD) with momentum 0.9
      - L1 weight decay: $5\times 10^{-4} \times 0.15$
      - L2 weight decay: $5\times 10^{-4} \times 0.85$
      - Learning rate scheduler $$lr(t)=\frac{1}{\alpha(t+t_0)}$$ where $t_0=\frac{1}{\alpha\times lr(0)},\alpha=0.001$, $lr(0)$ is the initial learning rate.
      - Initial learning rate (hard to determine manually)
          - ResNet18, ResNet34, MobileNetV2, VGG11
              - 0.1
          - GreenAINet
              - 0.005

## Comparison

|             | num_learn_params | num_learn_params | avg_secs_epoch | best_val_acc |
| ----------- | ---------------: | ---------------: | -------------: | -----------: |
| MobileNetV2 | 2.28M            | 12800            | 39.864         | 0.8176         |
| ResNet18    | 11.17M           | 5120             | 32.268         | 0.8062         |
| ResNet34    | 21.28M           | 5120             | 58.000         | 0.8223         |
| VGG11       | 9.23M            | 5120             | 25.824         | 0.7853         |
| gv856       | 0 (batch0: 0.1M) | 2.27M            | 221.08         | 0.8098         |
| gv972       | 0 (batch0: 0.34M)| 2.42M            | 137.23         | 0.8271         |
| ResNet101   | 42.49M           | 20480            | 244.11         | 0.8150         |
| VGG16       | 14.72M           | **18.91M**       | **51.15**      | **0.7988**     |

GAC 在 accuracy 上是跟其他 Deep CNN 實力相當的，最大的問題在於計算時間，一個 epoch 為 137 秒相當於 2 倍的 ResNet34。

## Architecture of ResNet34 & GreenAI(gv856.py)

  - ResNet34: 深, 狹窄
  - GreenAI: 淺, 寬
  
![](https://i.imgur.com/dlheAg0.jpg)
![](https://i.imgur.com/RyoyXB1.jpg)

## Bottleneck Layer (ResNet)
![](https://i.imgur.com/UR66gFZ.png)


## Benchmark (2019/07/16)

- L1 weight decay: $5\times 10^{-4} \times 0$
- L2 weight decay: $5\times 10^{-4} \times 1$

|          |      | num_learn_params | avg_secs_epoch  | best_val_acc  |
| -------- |------| ---------------: | --------------: | ------------: |
| gv972    |`p=2` | 2.76M            | 137.23          | 0.8271        |
| gv972    |`p=1` | 2.76M            | 138.84          | 0.8251        |
| gv856    |`p=2` | 2.37M            | 221.08          | 0.8106        |
| gv856    |`p=1` | 2.37M            | 219.36          | 0.8031        |

## Benchmark (2019/07/24 MTL data)

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       | 80.002         | 0.8506       |
| ResNet18    | 22.586         | 0.8584       |
| ResNet34    | 36.689         | 0.8543       |
| MobileNetV2 | 26.841         | 0.8589       |
| VGG11       | 18.770         | 0.8559       |

## Benchmark (2019/08/13 MTL data - good/FM)
data directory: ~/alfos/benchmark/MTL_data/data/

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       | 26.598         | 0.9838       |
| ResNet18    |                | 0.9878       |
| ResNet34    | 12.321         | 0.9895       |
| MobileNetV2 |          |        |
| VGG11       |          |        |

gv972 confusion matrix
    
    [[1979    9]
     [  27  280]]
     
ResNet34 confusion matrix

    [[1985    3]
     [  20  287]]
     
data directory: ~/alfos/benchmark/MTL_data/data/data2

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       |          | 0.9160       |
| ResNet18    |          | 0.9878       |
| ResNet34    |          | 0.9895       |
| MobileNetV2 |          |        |
| VGG11       |          |        |

gv972 

    [[1149   25]
     [ 144 1030]]
     
## Benchmark (2019/08/13 MTL data - good/FM/Scratch_Gold)
### data directory: ~/alfos/benchmark/MTL_data/data/data3

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       | 18.665         | 0.9205       |
| ResNet18    | 10.977         | 0.9673       |
| ResNet34    |          |        |
| MobileNetV2 |          |        |
| VGG11       |          |        |

gv972

    [[7003  215  319]
     [  25   16    9]
     [  35    7   51]]
     
    Recall: [0.9291495289903144, 0.32, 0.5483870967741935]
    Precision: [0.9915050261928359, 0.06722689075630252, 0.1345646437994723]
     
ResNet18

    [[7358   35  144]
     [  29   13    8]
     [  31    0   62]]
    
    Recall: [0.9762504975454425, 0.26, 0.6666666666666666]
    Precision: [0.9919115664599623, 0.2708333333333333, 0.2897196261682243]
    
### data directory: ~/alfos/benchmark/MTL_data/data/data4

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       | 33.168         | 0.9388       |
| ResNet18    | 12.316         | 0.9773       |
| ResNet34    |          |        |
| MobileNetV2 |          |        |
| VGG11       |          |        |

gv972

    [[7107  118  271]
     [  38   18    6]
     [  34    3   85]]
     
    Recall: [0.9481056563500534, 0.2903225806451613, 0.6967213114754098]
    Precision: [0.9899707480150439, 0.12949640287769784, 0.23480662983425415]
     
ResNet18

    [[7410   20   66]
     [  43   18    1]
     [  43    0   79]]
     
    Recall: [0.9885272145144077, 0.2903225806451613, 0.6475409836065574]
    Precision: [0.9885272145144077, 0.47368421052631576, 0.541095890410959]
    
### data directory: ~/alfos/benchmark/MTL_data/data/data5

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       | 38.857         | 0.7880       |
| ResNet18    | 14.217         | 0.8675       |
| ResNet34    |          |        |
| MobileNetV2 |          |        |
| VGG11       |          |        |

gv972

    [[5779  926  451]
     [  53   76   43]
     [ 116   39  197]]
     
    Recall: [0.9963666852990497, 0.0, 0.28125]
    Precision: [0.9481382978723404, 0.0, 0.6226415094339622]
     
ResNet18

    [[6745   51  360]
     [  95   18   59]
     [ 148   14  190]]
     
    Recall: [0.9425656791503634, 0.10465116279069768, 0.5397727272727273]
    Precision: [0.9652261018889525, 0.21686746987951808, 0.31198686371100165]
    
## Benchmark (2019/08/20 MTL data)

source:
 - training: `/data/spd/ash/FM_benchmarking_6000/training_files`
 - testing: `/data/spd/ash/FM_benchmarking_6000/testing_files/image_partition`


|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       |          |        |
| ResNet18    | NA             | 0.8516       |
| ResNet101   | NA             | 0.8435       |
| VGG16       | NA             | 0.8460       |


ResNet18

    [[7612  233]
     [1957 5888]]
     
    Recall: [0.9702995538559592, 0.7505417463352454]
    Precision: [0.7954854216741561, 0.9619343244567881]
    
ResNet101

    [[7673  172]
     [2275 5570]]
     
    Recall: [0.9780752071383046, 0.710006373486297]
    Precision: [0.7713108162444713, 0.970045280390108]
    
VGG16

    [[7480  365]
     [1915 5930]]
     
    Recall: [0.9534735500318674, 0.7558954748247291]
    Precision: [0.7961681745609367, 0.9420174741858618]
---
source FMSM
 - training: `/data/spd/ash/FMSM_benchmarking_3000/training_files`
 - testing: `/data/spd/ash/FMSM_benchmarking_3000/testing_files/image_partition`

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       |          |        |
| ResNet18    | NA             | 0.7566       |
| ResNet101   | NA             | 0.7628       |
| VGG16       | NA             | 0.7642       |

ResNet18

    [[10236  1463]
     [ 3914  7785]]
     
    Recall: [0.8749465766304813, 0.665441490725703]
    Precision: [0.7233922261484099, 0.8418036332179931]
    
ResNet101

    [[10631  1068]
     [ 4338  7361]]
     
    Recall: [0.908710146166339, 0.6291990768441748]
    Precision: [0.7102010822366224, 0.8732945782417844]

VGG16
    
    [[10541  1158]
     [ 4123  7576]]
     
    Recall: [0.9010171809556372, 0.647576715958629]
    Precision: [0.718835242771413, 0.8674147011678498]

---
data8 - source SMScratch
 - training: `/data/spd/ash/SMScratch_benchmarking_1200/training_files`
 - testing: `/data/spd/ash/SMScratch_benchmarking_1200/testing_files/image_partition`

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       |          |        |
| ResNet18    | NA             | 0.7237       |
| ResNet101   | NA             | 0.7519       |
| VGG16       | NA             | 0.7403       |

ResNet18

    [[8826 1261]
     [3929 6250]]
     
    Recall: [0.8749876078120353, 0.6140092346988899]
    Precision: [0.6919639357114857, 0.8321129010784183]
ResNet101

    [[8470 1617]
     [3402 6777]]
    
    Recall: [0.8396946564885496, 0.6657824933687002]
    Precision: [0.7134433962264151, 0.8073624017155111]

VGG16

    [[9170  917]
     [4169 6010]]
     
    Recall: [0.9090909090909091, 0.5904312800864525]
    Precision: [0.6874578304220706, 0.8676194600837304]
    
---
data9 - source Scratch_Gold
 - training: `/data/spd/ash/Scratch_Gold_benchmarking_2500/training_files`
 - testing: `/data/spd/ash/Scratch_Gold_benchmarking_2500/testing_files/image_partition`

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       |          |        |
| ResNet18    | NA             | 0.8393       |
| ResNet101   | NA             | 0.8511       |
| VGG16       | NA             | 0.8445       |

ResNet18

    [[5959  535]
     [1247 5247]]
    
    Recall: [0.9176162611641515, 0.8079765937788728]
    Precision: [0.8269497640854844, 0.9074714631615358]
    
ResNet101

    [[6229  265]
     [1593 4901]]
     
    Recall: [0.9591931013242994, 0.7546966430551278]
    Precision: [0.7963436461263104, 0.948703058459156]

VGG16

    [[6086  408]
     [1372 5122]]
     
    Recall: [0.93717277486911, 0.7887280566676933]
    Precision: [0.8160364709037276, 0.9262206148282097]
    
---
data10 - source all 4 class
 - training: `/data/spd/ash/all_benchmarking_None/training_files`
 - testing: `/data/spd/ash/all_benchmarking_None/testing_files/image_partition`

|             | avg_secs_epoch | best_val_acc |
| ----------- | --------------:| ------------:|
| gv972       |          |        |
| ResNet18    | NA             | 0.6114       |
| ResNet101   | NA             | 0.5814       |
| VGG16       | NA             | 0.6122       |

ResNet18

    [[24253   923  1087  5390  4456]
     [ 1830  5019   946    30    12]
     [ 1303   498  4678     8     7]
     [ 3784    84    45  6216  1512]
     [ 2362    11     6  3017  4833]]
     
    Recall: [0.6716608047855105, 0.6404236314916422, 0.7203572528487835, 0.533974744437763, 0.4724802033434353]
    Precision: [0.723279255636407, 0.768018362662586, 0.6918071576456669, 0.4239819930427665, 0.4466728280961183]
    
ResNet101

    [[19814  1162  1418  6012  7703]
     [ 1561  5258   924    59    35]
     [ 1011   500  4949    17    17]
     [ 1945    36    51  6149  3460]
     [ 1396     6    11  1812  7004]]
     
    Recall: [0.5487274640671301, 0.6709199948960062, 0.7620880813058207, 0.5282192251524783, 0.684719913970085]
    Precision: [0.7701636413106853, 0.7552427463372594, 0.6730586155310757, 0.43768239732365294, 0.3844338328118997]
    
VGG16

    [[23568  1119  1615  5186  4621]
     [ 1624  5341   821    35    16]
     [  938   466  5072     6    12]
     [ 2790   123    75  6543  2110]
     [ 2186    36    51  1971  5985]]
     
    Recall: [0.6526904649810297, 0.6815107821870614, 0.7810286418232214, 0.5620651146808694, 0.5851011829113305]
    Precision: [0.7576673310615315, 0.7538461538461538, 0.6643961226093791, 0.4761662178880722, 0.4696327683615819]