---
# System prepended metadata

title: Fault Tolerant RRAM-Based Neuromorphic Computing with Static and Dynamic Variations
tags: [paper]

---

# Fault Tolerant RRAM-Based Neuromorphic Computing with Static and Dynamic Variations

###### tags: `paper`

[TOC]

## 12/17
### goal : traicing `main.py`
* [source](https://hackmd.io/@hsieh22/fault_tolerant_rram_src1)

### function and purpose
* parser
    * function : define parameter and control program by input
    * each parameter has clearly defined in src (in help field)

* class
    * quantize_opts
        * default mode is `none`, width is `-1`, base is `-1`
        * user can choose quantize or not
    
    * LeNet

* case situation:
    * decide which structure should be used, there are eight modules
    * because this work solve static variation through adding a new layer after normal input to adjust weight, so we have to modified original structure, the followong eight structures use in this work.
    * resnet18
    * resnet34
    * mobilenet
    * lenet
    * inputmap_vgg11_bn - specific module to fit this work
    * inputmap_vgg16_bn - specific module to fit this work
    * inputmap_resnet18 - specific module to fit this work
    * inputmap_resnet34 - specific module to fit this work

* CUDA
    * what is CUDA ? Excerpt from [CUDA](https://blogs.nvidia.com/blog/2012/09/10/what-is-cuda-2/)

> CUDA is a parallel computing platform and programming model that makes using a GPU for general purpose computing simple and elegant. The developer still programs in the familiar C, C++, Fortran, or an ever expanding list of supported languages, and incorporates extensions of these languages in the form of a few basic keywords.


## 12/10

### environment setting
* deploy on `140.116.245.115`
```
Linux gpuserval-System-Product-Name 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
```
* python version ( default python version is 2.6, we have to change to 3.6 through adding alias )
```
Python 3.6.9
```

* create python virtual environment
```
$apt-get update
$apt-get install python-virtualenv
$virtualenv -p /usr/bin/python3 virtualenv/project_name
$cd virtualenv/project_name/bin
$source activate
```

* install tool, check this [website](https://pytorch.org/) to get suitable version and install
* check CUDA version by `nvidia-sim`
```
pip install torch==1.7.0+cu101 torchvision==0.8.1+cu101 torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
```

* then we can run `main.py`
```
python main.py
```

training result

:::spoiler

```
(Epoch - 0) Accuracy on Trainset: 36.08 %
(Epoch - 0) Accuracy on Testset: 49.57 %
(Epoch - 1) Accuracy on Trainset: 55.62 %
(Epoch - 1) Accuracy on Testset: 62.31 %
(Epoch - 2) Accuracy on Trainset: 64.38 %
(Epoch - 2) Accuracy on Testset: 62.48 %
(Epoch - 3) Accuracy on Trainset: 69.50 %
(Epoch - 3) Accuracy on Testset: 67.22 %
(Epoch - 4) Accuracy on Trainset: 73.15 %
(Epoch - 4) Accuracy on Testset: 67.53 %
(Epoch - 5) Accuracy on Trainset: 75.67 %
(Epoch - 5) Accuracy on Testset: 74.35 %
(Epoch - 6) Accuracy on Trainset: 77.86 %
(Epoch - 6) Accuracy on Testset: 74.34 %
(Epoch - 7) Accuracy on Trainset: 79.13 %
(Epoch - 7) Accuracy on Testset: 78.17 %
(Epoch - 8) Accuracy on Trainset: 80.77 %
(Epoch - 8) Accuracy on Testset: 77.79 %
(Epoch - 9) Accuracy on Trainset: 81.57 %
(Epoch - 9) Accuracy on Testset: 77.85 %
(Epoch - 10) Accuracy on Trainset: 82.97 %
(Epoch - 10) Accuracy on Testset: 78.25 %
(Epoch - 11) Accuracy on Trainset: 83.63 %
(Epoch - 11) Accuracy on Testset: 79.39 %
(Epoch - 12) Accuracy on Trainset: 84.54 %
(Epoch - 12) Accuracy on Testset: 80.11 %
(Epoch - 13) Accuracy on Trainset: 85.28 %
(Epoch - 13) Accuracy on Testset: 80.85 %
(Epoch - 14) Accuracy on Trainset: 85.90 %
(Epoch - 14) Accuracy on Testset: 79.88 %
(Epoch - 15) Accuracy on Trainset: 86.47 %
(Epoch - 15) Accuracy on Testset: 82.08 %
(Epoch - 16) Accuracy on Trainset: 87.16 %
(Epoch - 16) Accuracy on Testset: 84.93 %
(Epoch - 17) Accuracy on Trainset: 87.41 %
(Epoch - 17) Accuracy on Testset: 84.79 %
(Epoch - 18) Accuracy on Trainset: 87.83 %
(Epoch - 18) Accuracy on Testset: 84.20 %
(Epoch - 19) Accuracy on Trainset: 88.03 %
(Epoch - 19) Accuracy on Testset: 83.65 %
(Epoch - 20) Accuracy on Trainset: 88.40 %
(Epoch - 20) Accuracy on Testset: 81.61 %
(Epoch - 21) Accuracy on Trainset: 88.51 %
(Epoch - 21) Accuracy on Testset: 83.12 %
(Epoch - 22) Accuracy on Trainset: 89.12 %
(Epoch - 22) Accuracy on Testset: 82.92 %
(Epoch - 23) Accuracy on Trainset: 89.04 %
(Epoch - 23) Accuracy on Testset: 84.11 %
(Epoch - 24) Accuracy on Trainset: 89.53 %
(Epoch - 24) Accuracy on Testset: 82.76 %
(Epoch - 25) Accuracy on Trainset: 89.45 %
(Epoch - 25) Accuracy on Testset: 85.83 %
(Epoch - 26) Accuracy on Trainset: 89.76 %
(Epoch - 26) Accuracy on Testset: 81.08 %
(Epoch - 27) Accuracy on Trainset: 89.91 %
(Epoch - 27) Accuracy on Testset: 83.48 %
(Epoch - 28) Accuracy on Trainset: 90.09 %
(Epoch - 28) Accuracy on Testset: 84.74 %
(Epoch - 29) Accuracy on Trainset: 90.42 %
(Epoch - 29) Accuracy on Testset: 84.26 %
(Epoch - 30) Accuracy on Trainset: 93.42 %
(Epoch - 30) Accuracy on Testset: 88.66 %
(Epoch - 31) Accuracy on Trainset: 94.09 %
(Epoch - 31) Accuracy on Testset: 88.67 %
(Epoch - 32) Accuracy on Trainset: 94.48 %
(Epoch - 32) Accuracy on Testset: 87.32 %
(Epoch - 33) Accuracy on Trainset: 94.40 %
(Epoch - 33) Accuracy on Testset: 88.09 %
(Epoch - 34) Accuracy on Trainset: 94.57 %
(Epoch - 34) Accuracy on Testset: 87.50 %
(Epoch - 35) Accuracy on Trainset: 94.71 %
(Epoch - 35) Accuracy on Testset: 87.54 %
(Epoch - 36) Accuracy on Trainset: 94.60 %
(Epoch - 36) Accuracy on Testset: 85.94 %
(Epoch - 37) Accuracy on Trainset: 94.80 %
(Epoch - 37) Accuracy on Testset: 86.72 %
(Epoch - 38) Accuracy on Trainset: 94.77 %
(Epoch - 38) Accuracy on Testset: 87.06 %
(Epoch - 39) Accuracy on Trainset: 94.73 %
(Epoch - 39) Accuracy on Testset: 87.23 %
(Epoch - 40) Accuracy on Trainset: 94.74 %
(Epoch - 40) Accuracy on Testset: 88.06 %
(Epoch - 41) Accuracy on Trainset: 94.83 %
(Epoch - 41) Accuracy on Testset: 87.52 %
(Epoch - 42) Accuracy on Trainset: 94.78 %
(Epoch - 42) Accuracy on Testset: 87.10 %
(Epoch - 43) Accuracy on Trainset: 95.01 %
(Epoch - 43) Accuracy on Testset: 87.01 %
(Epoch - 44) Accuracy on Trainset: 95.08 %
(Epoch - 44) Accuracy on Testset: 86.77 %
(Epoch - 45) Accuracy on Trainset: 94.91 %
(Epoch - 45) Accuracy on Testset: 87.48 %
(Epoch - 46) Accuracy on Trainset: 95.18 %
(Epoch - 46) Accuracy on Testset: 86.14 %
(Epoch - 47) Accuracy on Trainset: 95.25 %
(Epoch - 47) Accuracy on Testset: 88.04 %
(Epoch - 48) Accuracy on Trainset: 95.09 %
(Epoch - 48) Accuracy on Testset: 86.95 %
(Epoch - 49) Accuracy on Trainset: 95.06 %
(Epoch - 49) Accuracy on Testset: 87.39 %
(Epoch - 50) Accuracy on Trainset: 95.22 %
(Epoch - 50) Accuracy on Testset: 87.44 %
(Epoch - 51) Accuracy on Trainset: 95.15 %
(Epoch - 51) Accuracy on Testset: 87.30 %
(Epoch - 52) Accuracy on Trainset: 95.22 %
(Epoch - 52) Accuracy on Testset: 84.95 %
(Epoch - 53) Accuracy on Trainset: 95.23 %
(Epoch - 53) Accuracy on Testset: 87.74 %
(Epoch - 54) Accuracy on Trainset: 95.36 %
(Epoch - 54) Accuracy on Testset: 87.48 %
(Epoch - 55) Accuracy on Trainset: 95.59 %
(Epoch - 55) Accuracy on Testset: 87.51 %
(Epoch - 56) Accuracy on Trainset: 95.26 %
(Epoch - 56) Accuracy on Testset: 87.96 %
(Epoch - 57) Accuracy on Trainset: 95.46 %
(Epoch - 57) Accuracy on Testset: 87.81 %
(Epoch - 58) Accuracy on Trainset: 95.52 %
(Epoch - 58) Accuracy on Testset: 87.12 %
(Epoch - 59) Accuracy on Trainset: 95.82 %
(Epoch - 59) Accuracy on Testset: 86.89 %
(Epoch - 60) Accuracy on Trainset: 97.50 %
(Epoch - 60) Accuracy on Testset: 89.40 %
(Epoch - 61) Accuracy on Trainset: 97.95 %
(Epoch - 61) Accuracy on Testset: 89.83 %
(Epoch - 62) Accuracy on Trainset: 98.16 %
(Epoch - 62) Accuracy on Testset: 89.31 %
(Epoch - 63) Accuracy on Trainset: 98.23 %
(Epoch - 63) Accuracy on Testset: 89.20 %
(Epoch - 64) Accuracy on Trainset: 98.29 %
(Epoch - 64) Accuracy on Testset: 89.66 %
(Epoch - 65) Accuracy on Trainset: 98.39 %
(Epoch - 65) Accuracy on Testset: 89.55 %
(Epoch - 66) Accuracy on Trainset: 98.40 %
(Epoch - 66) Accuracy on Testset: 89.26 %
(Epoch - 67) Accuracy on Trainset: 98.29 %
(Epoch - 67) Accuracy on Testset: 88.83 %
(Epoch - 68) Accuracy on Trainset: 98.48 %
(Epoch - 68) Accuracy on Testset: 89.28 %
(Epoch - 69) Accuracy on Trainset: 98.39 %
(Epoch - 69) Accuracy on Testset: 89.37 %
(Epoch - 70) Accuracy on Trainset: 98.15 %
(Epoch - 70) Accuracy on Testset: 88.90 %
(Epoch - 71) Accuracy on Trainset: 98.30 %
(Epoch - 71) Accuracy on Testset: 88.76 %
(Epoch - 72) Accuracy on Trainset: 98.26 %
(Epoch - 72) Accuracy on Testset: 89.58 %
(Epoch - 73) Accuracy on Trainset: 98.20 %
(Epoch - 73) Accuracy on Testset: 88.38 %
(Epoch - 74) Accuracy on Trainset: 98.38 %
(Epoch - 74) Accuracy on Testset: 89.65 %
(Epoch - 75) Accuracy on Trainset: 98.27 %
(Epoch - 75) Accuracy on Testset: 88.73 %
(Epoch - 76) Accuracy on Trainset: 98.31 %
(Epoch - 76) Accuracy on Testset: 89.46 %
(Epoch - 77) Accuracy on Trainset: 98.15 %
(Epoch - 77) Accuracy on Testset: 89.32 %
(Epoch - 78) Accuracy on Trainset: 98.38 %
(Epoch - 78) Accuracy on Testset: 89.10 %
(Epoch - 79) Accuracy on Trainset: 98.12 %
(Epoch - 79) Accuracy on Testset: 88.19 %
(Epoch - 80) Accuracy on Trainset: 98.24 %
(Epoch - 80) Accuracy on Testset: 88.49 %
(Epoch - 81) Accuracy on Trainset: 98.17 %
(Epoch - 81) Accuracy on Testset: 89.28 %
(Epoch - 82) Accuracy on Trainset: 98.36 %
(Epoch - 82) Accuracy on Testset: 88.86 %
(Epoch - 83) Accuracy on Trainset: 98.11 %
(Epoch - 83) Accuracy on Testset: 88.58 %
(Epoch - 84) Accuracy on Trainset: 98.15 %
(Epoch - 84) Accuracy on Testset: 89.24 %
(Epoch - 85) Accuracy on Trainset: 98.09 %
(Epoch - 85) Accuracy on Testset: 89.21 %
(Epoch - 86) Accuracy on Trainset: 98.37 %
(Epoch - 86) Accuracy on Testset: 88.15 %
(Epoch - 87) Accuracy on Trainset: 97.94 %
(Epoch - 87) Accuracy on Testset: 88.48 %
(Epoch - 88) Accuracy on Trainset: 98.13 %
(Epoch - 88) Accuracy on Testset: 88.35 %
(Epoch - 89) Accuracy on Trainset: 98.03 %
(Epoch - 89) Accuracy on Testset: 88.21 %
(Epoch - 90) Accuracy on Trainset: 99.06 %
(Epoch - 90) Accuracy on Testset: 90.59 %
(Epoch - 91) Accuracy on Trainset: 99.32 %
(Epoch - 91) Accuracy on Testset: 90.55 %
(Epoch - 92) Accuracy on Trainset: 99.42 %
(Epoch - 92) Accuracy on Testset: 90.68 %
(Epoch - 93) Accuracy on Trainset: 99.45 %
(Epoch - 93) Accuracy on Testset: 90.68 %
(Epoch - 94) Accuracy on Trainset: 99.55 %
(Epoch - 94) Accuracy on Testset: 90.99 %
(Epoch - 95) Accuracy on Trainset: 99.56 %
(Epoch - 95) Accuracy on Testset: 90.68 %
(Epoch - 96) Accuracy on Trainset: 99.54 %
(Epoch - 96) Accuracy on Testset: 90.76 %
(Epoch - 97) Accuracy on Trainset: 99.51 %
(Epoch - 97) Accuracy on Testset: 90.73 %
(Epoch - 98) Accuracy on Trainset: 99.52 %
(Epoch - 98) Accuracy on Testset: 90.66 %
(Epoch - 99) Accuracy on Trainset: 99.52 %
(Epoch - 99) Accuracy on Testset: 90.89 %
(Epoch - 100) Accuracy on Trainset: 99.53 %
(Epoch - 100) Accuracy on Testset: 90.56 %
(Epoch - 101) Accuracy on Trainset: 99.48 %
(Epoch - 101) Accuracy on Testset: 90.43 %
(Epoch - 102) Accuracy on Trainset: 99.62 %
(Epoch - 102) Accuracy on Testset: 90.64 %
(Epoch - 103) Accuracy on Trainset: 99.56 %
(Epoch - 103) Accuracy on Testset: 90.67 %
(Epoch - 104) Accuracy on Trainset: 99.52 %
(Epoch - 104) Accuracy on Testset: 90.81 %
(Epoch - 105) Accuracy on Trainset: 99.57 %
(Epoch - 105) Accuracy on Testset: 91.04 %
(Epoch - 106) Accuracy on Trainset: 99.59 %
(Epoch - 106) Accuracy on Testset: 90.76 %
(Epoch - 107) Accuracy on Trainset: 99.53 %
(Epoch - 107) Accuracy on Testset: 90.29 %
(Epoch - 108) Accuracy on Trainset: 99.48 %
(Epoch - 108) Accuracy on Testset: 90.39 %
(Epoch - 109) Accuracy on Trainset: 99.56 %
(Epoch - 109) Accuracy on Testset: 90.59 %
(Epoch - 110) Accuracy on Trainset: 99.61 %
(Epoch - 110) Accuracy on Testset: 90.63 %
(Epoch - 111) Accuracy on Trainset: 99.65 %
(Epoch - 111) Accuracy on Testset: 90.69 %
(Epoch - 112) Accuracy on Trainset: 99.54 %
(Epoch - 112) Accuracy on Testset: 90.45 %
(Epoch - 113) Accuracy on Trainset: 99.59 %
(Epoch - 113) Accuracy on Testset: 91.06 %
(Epoch - 114) Accuracy on Trainset: 99.61 %
(Epoch - 114) Accuracy on Testset: 90.77 %
(Epoch - 115) Accuracy on Trainset: 99.67 %
(Epoch - 115) Accuracy on Testset: 90.42 %
(Epoch - 116) Accuracy on Trainset: 99.50 %
(Epoch - 116) Accuracy on Testset: 90.62 %
(Epoch - 117) Accuracy on Trainset: 99.58 %
(Epoch - 117) Accuracy on Testset: 90.37 %
(Epoch - 118) Accuracy on Trainset: 99.48 %
(Epoch - 118) Accuracy on Testset: 89.94 %
(Epoch - 119) Accuracy on Trainset: 99.54 %
(Epoch - 119) Accuracy on Testset: 90.51 %
(Epoch - 120) Accuracy on Trainset: 99.68 %
(Epoch - 120) Accuracy on Testset: 91.02 %
(Epoch - 121) Accuracy on Trainset: 99.79 %
(Epoch - 121) Accuracy on Testset: 91.31 %
(Epoch - 122) Accuracy on Trainset: 99.84 %
(Epoch - 122) Accuracy on Testset: 91.22 %
(Epoch - 123) Accuracy on Trainset: 99.84 %
(Epoch - 123) Accuracy on Testset: 91.40 %
(Epoch - 124) Accuracy on Trainset: 99.86 %
(Epoch - 124) Accuracy on Testset: 91.27 %
(Epoch - 125) Accuracy on Trainset: 99.89 %
(Epoch - 125) Accuracy on Testset: 91.22 %
(Epoch - 126) Accuracy on Trainset: 99.88 %
(Epoch - 126) Accuracy on Testset: 91.24 %
(Epoch - 127) Accuracy on Trainset: 99.93 %
(Epoch - 127) Accuracy on Testset: 91.50 %
(Epoch - 128) Accuracy on Trainset: 99.88 %
(Epoch - 128) Accuracy on Testset: 90.91 %
(Epoch - 129) Accuracy on Trainset: 99.89 %
(Epoch - 129) Accuracy on Testset: 91.29 %
(Epoch - 130) Accuracy on Trainset: 99.90 %
(Epoch - 130) Accuracy on Testset: 91.34 %
(Epoch - 131) Accuracy on Trainset: 99.92 %
(Epoch - 131) Accuracy on Testset: 91.43 %
(Epoch - 132) Accuracy on Trainset: 99.92 %
(Epoch - 132) Accuracy on Testset: 91.14 %
(Epoch - 133) Accuracy on Trainset: 99.92 %
(Epoch - 133) Accuracy on Testset: 91.38 %
(Epoch - 134) Accuracy on Trainset: 99.91 %
(Epoch - 134) Accuracy on Testset: 91.38 %
(Epoch - 135) Accuracy on Trainset: 99.94 %
(Epoch - 135) Accuracy on Testset: 91.61 %
(Epoch - 136) Accuracy on Trainset: 99.91 %
(Epoch - 136) Accuracy on Testset: 91.52 %
(Epoch - 137) Accuracy on Trainset: 99.96 %
(Epoch - 137) Accuracy on Testset: 91.74 %
(Epoch - 138) Accuracy on Trainset: 99.94 %
(Epoch - 138) Accuracy on Testset: 91.69 %
(Epoch - 139) Accuracy on Trainset: 99.92 %
(Epoch - 139) Accuracy on Testset: 91.42 %
(Epoch - 140) Accuracy on Trainset: 99.93 %
(Epoch - 140) Accuracy on Testset: 91.57 %
(Epoch - 141) Accuracy on Trainset: 99.91 %
(Epoch - 141) Accuracy on Testset: 91.43 %
(Epoch - 142) Accuracy on Trainset: 99.93 %
(Epoch - 142) Accuracy on Testset: 91.59 %
(Epoch - 143) Accuracy on Trainset: 99.94 %
(Epoch - 143) Accuracy on Testset: 91.56 %
(Epoch - 144) Accuracy on Trainset: 99.93 %
(Epoch - 144) Accuracy on Testset: 91.47 %
(Epoch - 145) Accuracy on Trainset: 99.95 %
(Epoch - 145) Accuracy on Testset: 91.64 %
(Epoch - 146) Accuracy on Trainset: 99.95 %
(Epoch - 146) Accuracy on Testset: 91.37 %
(Epoch - 147) Accuracy on Trainset: 99.95 %
(Epoch - 147) Accuracy on Testset: 91.66 %
(Epoch - 148) Accuracy on Trainset: 99.97 %
(Epoch - 148) Accuracy on Testset: 91.80 %
(Epoch - 149) Accuracy on Trainset: 99.96 %
(Epoch - 149) Accuracy on Testset: 91.58 %
(Epoch - 150) Accuracy on Trainset: 99.94 %
(Epoch - 150) Accuracy on Testset: 91.82 %
```

:::

---

## 11/26

### Source code structure

#### datasets
* cifar10data
    * cifar-10-batches-py
        * data_batch_1 ~data_batch_5
* mnist
    * MNIST
        * processed
            * `test.pt`
            * `training.pt`
        * raw
            * t10k-images-idx3-ubyte
            * t10k-labels-idx1-ubyte
            * train-images-idx3-ubyte
            * train-labels-idx1-ubyte
---

#### ResNet
* inputmap of ResNet (it's PT file)
* `main.py` : main program, parser is define here
* `models_resnet.py` : define the model of ResNet in this work, here we make ResNet18 for testing
* `models_resnet_inputmap.py`
* `models_vgg.py` : define the model of VGG in this work
* `models_vgg_inputmap.py`
* `utils.py` : define several funcs use in this work
* `run.sh` : run all file, just like makefile

---

#### VGG
* inputmap of VGG
* other files are same with files in ResNet
---

#### venv
* environment data

---- 

#### Code tracing

##### `main.py`

[source](https://hackmd.io/@hsieh22/fault_tolerant_rram_src1)

* define and implement parser
    * parser will define parameter how to control program, then it will reference parameters to run training, finetune and evaluation
    * we can set parser, resume (directory that storing weight),the bits of quantizing, network architecture or restoring data

* define quantize_opt
* define LeNet
* deal with different datasets
* inference step
* finetune step
* quantize options
* training


----

##### `utils.py`


* implement all details of function
    * Variation()
    * DynamicVariation()
    * quantize()
        * deal with three type of quantize : linear, exp and power
    * SaveWeights()
    * RowRestoring_Weights()
    * RowRestoring_Gradients()
    * Train()
    * Eval()

---

### TODO
:::info
1. realize the structure of this work
2. trace each process in `main.py`
3. tracing `utils.py`
:::
            
            
