# Benchmark on CIFAR10 (50k+10k) ## Table of Contents [TOC] ## Models - GreenAINet (porting from gv856.py) - [MobileNetV2](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/mobilenetv2.py#L40) - [ResNet18](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/resnet.py#L100) - [ResNet34](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/resnet.py#L103) - [VGG11](https://github.com/kuangliu/pytorch-cifar/blob/ab908327d44bf9b1d22cd333a4466e85083d3f21/models/vgg.py#L14) This benchmark adopts MobileNetV2, ResNet18, ResNet34 and VGG11 from this [github repo](https://github.com/kuangliu/pytorch-cifar). ## Environment & Configuration - HW/SW - Python3.6 - Pytorch1.0 - Graphic card: GeForce GTX 1080Ti on spd7 - Training Config - Use no Data Augmentation - 50 epochs - Loss function: [`MultiMarginLoss(p=2)`](https://pytorch.org/docs/stable/nn.html#multimarginloss) - Optimize method: [`SGD`](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD) with momentum 0.9 - L1 weight decay: $5\times 10^{-4} \times 0.15$ - L2 weight decay: $5\times 10^{-4} \times 0.85$ - Learning rate scheduler $$lr(t)=\frac{1}{\alpha(t+t_0)}$$ where $t_0=\frac{1}{\alpha\times lr(0)},\alpha=0.001$, $lr(0)$ is the initial learning rate. - Initial learning rate (hard to determine manually) - ResNet18, ResNet34, MobileNetV2, VGG11 - 0.1 - GreenAINet - 0.005 ## Comparison | | num_learn_params | num_learn_params | avg_secs_epoch | best_val_acc | | ----------- | ---------------: | ---------------: | -------------: | -----------: | | MobileNetV2 | 2.28M | 12800 | 39.864 | 0.8176 | | ResNet18 | 11.17M | 5120 | 32.268 | 0.8062 | | ResNet34 | 21.28M | 5120 | 58.000 | 0.8223 | | VGG11 | 9.23M | 5120 | 25.824 | 0.7853 | | gv856 | 0 (batch0: 0.1M) | 2.27M | 221.08 | 0.8098 | | gv972 | 0 (batch0: 0.34M)| 2.42M | 137.23 | 0.8271 | | ResNet101 | 42.49M | 20480 | 244.11 | 0.8150 | | VGG16 | 14.72M | **18.91M** | **51.15** | **0.7988** | GAC 在 accuracy 上是跟其他 Deep CNN 實力相當的，最大的問題在於計算時間，一個 epoch 為 137 秒相當於 2 倍的 ResNet34。 ## Architecture of ResNet34 & GreenAI(gv856.py) - ResNet34: 深, 狹窄 - GreenAI: 淺, 寬 ![](https://i.imgur.com/dlheAg0.jpg) ![](https://i.imgur.com/RyoyXB1.jpg) ## Bottleneck Layer (ResNet) ![](https://i.imgur.com/UR66gFZ.png) ## Benchmark (2019/07/16) - L1 weight decay: $5\times 10^{-4} \times 0$ - L2 weight decay: $5\times 10^{-4} \times 1$ | | | num_learn_params | avg_secs_epoch | best_val_acc | | -------- |------| ---------------: | --------------: | ------------: | | gv972 |`p=2` | 2.76M | 137.23 | 0.8271 | | gv972 |`p=1` | 2.76M | 138.84 | 0.8251 | | gv856 |`p=2` | 2.37M | 221.08 | 0.8106 | | gv856 |`p=1` | 2.37M | 219.36 | 0.8031 | ## Benchmark (2019/07/24 MTL data) | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | 80.002 | 0.8506 | | ResNet18 | 22.586 | 0.8584 | | ResNet34 | 36.689 | 0.8543 | | MobileNetV2 | 26.841 | 0.8589 | | VGG11 | 18.770 | 0.8559 | ## Benchmark (2019/08/13 MTL data - good/FM) data directory: ~/alfos/benchmark/MTL_data/data/ | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | 26.598 | 0.9838 | | ResNet18 | | 0.9878 | | ResNet34 | 12.321 | 0.9895 | | MobileNetV2 | | | | VGG11 | | | gv972 confusion matrix [[1979 9] [ 27 280]] ResNet34 confusion matrix [[1985 3] [ 20 287]] data directory: ~/alfos/benchmark/MTL_data/data/data2 | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | | 0.9160 | | ResNet18 | | 0.9878 | | ResNet34 | | 0.9895 | | MobileNetV2 | | | | VGG11 | | | gv972 [[1149 25] [ 144 1030]] ## Benchmark (2019/08/13 MTL data - good/FM/Scratch_Gold) ### data directory: ~/alfos/benchmark/MTL_data/data/data3 | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | 18.665 | 0.9205 | | ResNet18 | 10.977 | 0.9673 | | ResNet34 | | | | MobileNetV2 | | | | VGG11 | | | gv972 [[7003 215 319] [ 25 16 9] [ 35 7 51]] Recall: [0.9291495289903144, 0.32, 0.5483870967741935] Precision: [0.9915050261928359, 0.06722689075630252, 0.1345646437994723] ResNet18 [[7358 35 144] [ 29 13 8] [ 31 0 62]] Recall: [0.9762504975454425, 0.26, 0.6666666666666666] Precision: [0.9919115664599623, 0.2708333333333333, 0.2897196261682243] ### data directory: ~/alfos/benchmark/MTL_data/data/data4 | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | 33.168 | 0.9388 | | ResNet18 | 12.316 | 0.9773 | | ResNet34 | | | | MobileNetV2 | | | | VGG11 | | | gv972 [[7107 118 271] [ 38 18 6] [ 34 3 85]] Recall: [0.9481056563500534, 0.2903225806451613, 0.6967213114754098] Precision: [0.9899707480150439, 0.12949640287769784, 0.23480662983425415] ResNet18 [[7410 20 66] [ 43 18 1] [ 43 0 79]] Recall: [0.9885272145144077, 0.2903225806451613, 0.6475409836065574] Precision: [0.9885272145144077, 0.47368421052631576, 0.541095890410959] ### data directory: ~/alfos/benchmark/MTL_data/data/data5 | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | 38.857 | 0.7880 | | ResNet18 | 14.217 | 0.8675 | | ResNet34 | | | | MobileNetV2 | | | | VGG11 | | | gv972 [[5779 926 451] [ 53 76 43] [ 116 39 197]] Recall: [0.9963666852990497, 0.0, 0.28125] Precision: [0.9481382978723404, 0.0, 0.6226415094339622] ResNet18 [[6745 51 360] [ 95 18 59] [ 148 14 190]] Recall: [0.9425656791503634, 0.10465116279069768, 0.5397727272727273] Precision: [0.9652261018889525, 0.21686746987951808, 0.31198686371100165] ## Benchmark (2019/08/20 MTL data) source: - training: `/data/spd/ash/FM_benchmarking_6000/training_files` - testing: `/data/spd/ash/FM_benchmarking_6000/testing_files/image_partition` | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | | | | ResNet18 | NA | 0.8516 | | ResNet101 | NA | 0.8435 | | VGG16 | NA | 0.8460 | ResNet18 [[7612 233] [1957 5888]] Recall: [0.9702995538559592, 0.7505417463352454] Precision: [0.7954854216741561, 0.9619343244567881] ResNet101 [[7673 172] [2275 5570]] Recall: [0.9780752071383046, 0.710006373486297] Precision: [0.7713108162444713, 0.970045280390108] VGG16 [[7480 365] [1915 5930]] Recall: [0.9534735500318674, 0.7558954748247291] Precision: [0.7961681745609367, 0.9420174741858618] --- source FMSM - training: `/data/spd/ash/FMSM_benchmarking_3000/training_files` - testing: `/data/spd/ash/FMSM_benchmarking_3000/testing_files/image_partition` | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | | | | ResNet18 | NA | 0.7566 | | ResNet101 | NA | 0.7628 | | VGG16 | NA | 0.7642 | ResNet18 [[10236 1463] [ 3914 7785]] Recall: [0.8749465766304813, 0.665441490725703] Precision: [0.7233922261484099, 0.8418036332179931] ResNet101 [[10631 1068] [ 4338 7361]] Recall: [0.908710146166339, 0.6291990768441748] Precision: [0.7102010822366224, 0.8732945782417844] VGG16 [[10541 1158] [ 4123 7576]] Recall: [0.9010171809556372, 0.647576715958629] Precision: [0.718835242771413, 0.8674147011678498] --- data8 - source SMScratch - training: `/data/spd/ash/SMScratch_benchmarking_1200/training_files` - testing: `/data/spd/ash/SMScratch_benchmarking_1200/testing_files/image_partition` | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | | | | ResNet18 | NA | 0.7237 | | ResNet101 | NA | 0.7519 | | VGG16 | NA | 0.7403 | ResNet18 [[8826 1261] [3929 6250]] Recall: [0.8749876078120353, 0.6140092346988899] Precision: [0.6919639357114857, 0.8321129010784183] ResNet101 [[8470 1617] [3402 6777]] Recall: [0.8396946564885496, 0.6657824933687002] Precision: [0.7134433962264151, 0.8073624017155111] VGG16 [[9170 917] [4169 6010]] Recall: [0.9090909090909091, 0.5904312800864525] Precision: [0.6874578304220706, 0.8676194600837304] --- data9 - source Scratch_Gold - training: `/data/spd/ash/Scratch_Gold_benchmarking_2500/training_files` - testing: `/data/spd/ash/Scratch_Gold_benchmarking_2500/testing_files/image_partition` | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | | | | ResNet18 | NA | 0.8393 | | ResNet101 | NA | 0.8511 | | VGG16 | NA | 0.8445 | ResNet18 [[5959 535] [1247 5247]] Recall: [0.9176162611641515, 0.8079765937788728] Precision: [0.8269497640854844, 0.9074714631615358] ResNet101 [[6229 265] [1593 4901]] Recall: [0.9591931013242994, 0.7546966430551278] Precision: [0.7963436461263104, 0.948703058459156] VGG16 [[6086 408] [1372 5122]] Recall: [0.93717277486911, 0.7887280566676933] Precision: [0.8160364709037276, 0.9262206148282097] --- data10 - source all 4 class - training: `/data/spd/ash/all_benchmarking_None/training_files` - testing: `/data/spd/ash/all_benchmarking_None/testing_files/image_partition` | | avg_secs_epoch | best_val_acc | | ----------- | --------------:| ------------:| | gv972 | | | | ResNet18 | NA | 0.6114 | | ResNet101 | NA | 0.5814 | | VGG16 | NA | 0.6122 | ResNet18 [[24253 923 1087 5390 4456] [ 1830 5019 946 30 12] [ 1303 498 4678 8 7] [ 3784 84 45 6216 1512] [ 2362 11 6 3017 4833]] Recall: [0.6716608047855105, 0.6404236314916422, 0.7203572528487835, 0.533974744437763, 0.4724802033434353] Precision: [0.723279255636407, 0.768018362662586, 0.6918071576456669, 0.4239819930427665, 0.4466728280961183] ResNet101 [[19814 1162 1418 6012 7703] [ 1561 5258 924 59 35] [ 1011 500 4949 17 17] [ 1945 36 51 6149 3460] [ 1396 6 11 1812 7004]] Recall: [0.5487274640671301, 0.6709199948960062, 0.7620880813058207, 0.5282192251524783, 0.684719913970085] Precision: [0.7701636413106853, 0.7552427463372594, 0.6730586155310757, 0.43768239732365294, 0.3844338328118997] VGG16 [[23568 1119 1615 5186 4621] [ 1624 5341 821 35 16] [ 938 466 5072 6 12] [ 2790 123 75 6543 2110] [ 2186 36 51 1971 5985]] Recall: [0.6526904649810297, 0.6815107821870614, 0.7810286418232214, 0.5620651146808694, 0.5851011829113305] Precision: [0.7576673310615315, 0.7538461538461538, 0.6643961226093791, 0.4761662178880722, 0.4696327683615819]