Fault Tolerant RRAM-Based Neuromorphic Computing with Static and Dynamic Variations

# Fault Tolerant RRAM-Based Neuromorphic Computing with Static and Dynamic Variations ###### tags: `paper` [TOC] ## 12/17 ### goal : traicing `main.py` * [source](https://hackmd.io/@hsieh22/fault_tolerant_rram_src1) ### function and purpose * parser * function : define parameter and control program by input * each parameter has clearly defined in src (in help field) * class * quantize_opts * default mode is `none`, width is `-1`, base is `-1` * user can choose quantize or not * LeNet * case situation: * decide which structure should be used, there are eight modules * because this work solve static variation through adding a new layer after normal input to adjust weight, so we have to modified original structure, the followong eight structures use in this work. * resnet18 * resnet34 * mobilenet * lenet * inputmap_vgg11_bn - specific module to fit this work * inputmap_vgg16_bn - specific module to fit this work * inputmap_resnet18 - specific module to fit this work * inputmap_resnet34 - specific module to fit this work * CUDA * what is CUDA ? Excerpt from [CUDA](https://blogs.nvidia.com/blog/2012/09/10/what-is-cuda-2/) > CUDA is a parallel computing platform and programming model that makes using a GPU for general purpose computing simple and elegant. The developer still programs in the familiar C, C++, Fortran, or an ever expanding list of supported languages, and incorporates extensions of these languages in the form of a few basic keywords. ## 12/10 ### environment setting * deploy on `140.116.245.115` ``` Linux gpuserval-System-Product-Name 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ``` * python version ( default python version is 2.6, we have to change to 3.6 through adding alias ) ``` Python 3.6.9 ``` * create python virtual environment ``` $apt-get update $apt-get install python-virtualenv $virtualenv -p /usr/bin/python3 virtualenv/project_name $cd virtualenv/project_name/bin $source activate ``` * install tool, check this [website](https://pytorch.org/) to get suitable version and install * check CUDA version by `nvidia-sim` ``` pip install torch==1.7.0+cu101 torchvision==0.8.1+cu101 torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html ``` * then we can run `main.py` ``` python main.py ``` training result :::spoiler ``` (Epoch - 0) Accuracy on Trainset: 36.08 % (Epoch - 0) Accuracy on Testset: 49.57 % (Epoch - 1) Accuracy on Trainset: 55.62 % (Epoch - 1) Accuracy on Testset: 62.31 % (Epoch - 2) Accuracy on Trainset: 64.38 % (Epoch - 2) Accuracy on Testset: 62.48 % (Epoch - 3) Accuracy on Trainset: 69.50 % (Epoch - 3) Accuracy on Testset: 67.22 % (Epoch - 4) Accuracy on Trainset: 73.15 % (Epoch - 4) Accuracy on Testset: 67.53 % (Epoch - 5) Accuracy on Trainset: 75.67 % (Epoch - 5) Accuracy on Testset: 74.35 % (Epoch - 6) Accuracy on Trainset: 77.86 % (Epoch - 6) Accuracy on Testset: 74.34 % (Epoch - 7) Accuracy on Trainset: 79.13 % (Epoch - 7) Accuracy on Testset: 78.17 % (Epoch - 8) Accuracy on Trainset: 80.77 % (Epoch - 8) Accuracy on Testset: 77.79 % (Epoch - 9) Accuracy on Trainset: 81.57 % (Epoch - 9) Accuracy on Testset: 77.85 % (Epoch - 10) Accuracy on Trainset: 82.97 % (Epoch - 10) Accuracy on Testset: 78.25 % (Epoch - 11) Accuracy on Trainset: 83.63 % (Epoch - 11) Accuracy on Testset: 79.39 % (Epoch - 12) Accuracy on Trainset: 84.54 % (Epoch - 12) Accuracy on Testset: 80.11 % (Epoch - 13) Accuracy on Trainset: 85.28 % (Epoch - 13) Accuracy on Testset: 80.85 % (Epoch - 14) Accuracy on Trainset: 85.90 % (Epoch - 14) Accuracy on Testset: 79.88 % (Epoch - 15) Accuracy on Trainset: 86.47 % (Epoch - 15) Accuracy on Testset: 82.08 % (Epoch - 16) Accuracy on Trainset: 87.16 % (Epoch - 16) Accuracy on Testset: 84.93 % (Epoch - 17) Accuracy on Trainset: 87.41 % (Epoch - 17) Accuracy on Testset: 84.79 % (Epoch - 18) Accuracy on Trainset: 87.83 % (Epoch - 18) Accuracy on Testset: 84.20 % (Epoch - 19) Accuracy on Trainset: 88.03 % (Epoch - 19) Accuracy on Testset: 83.65 % (Epoch - 20) Accuracy on Trainset: 88.40 % (Epoch - 20) Accuracy on Testset: 81.61 % (Epoch - 21) Accuracy on Trainset: 88.51 % (Epoch - 21) Accuracy on Testset: 83.12 % (Epoch - 22) Accuracy on Trainset: 89.12 % (Epoch - 22) Accuracy on Testset: 82.92 % (Epoch - 23) Accuracy on Trainset: 89.04 % (Epoch - 23) Accuracy on Testset: 84.11 % (Epoch - 24) Accuracy on Trainset: 89.53 % (Epoch - 24) Accuracy on Testset: 82.76 % (Epoch - 25) Accuracy on Trainset: 89.45 % (Epoch - 25) Accuracy on Testset: 85.83 % (Epoch - 26) Accuracy on Trainset: 89.76 % (Epoch - 26) Accuracy on Testset: 81.08 % (Epoch - 27) Accuracy on Trainset: 89.91 % (Epoch - 27) Accuracy on Testset: 83.48 % (Epoch - 28) Accuracy on Trainset: 90.09 % (Epoch - 28) Accuracy on Testset: 84.74 % (Epoch - 29) Accuracy on Trainset: 90.42 % (Epoch - 29) Accuracy on Testset: 84.26 % (Epoch - 30) Accuracy on Trainset: 93.42 % (Epoch - 30) Accuracy on Testset: 88.66 % (Epoch - 31) Accuracy on Trainset: 94.09 % (Epoch - 31) Accuracy on Testset: 88.67 % (Epoch - 32) Accuracy on Trainset: 94.48 % (Epoch - 32) Accuracy on Testset: 87.32 % (Epoch - 33) Accuracy on Trainset: 94.40 % (Epoch - 33) Accuracy on Testset: 88.09 % (Epoch - 34) Accuracy on Trainset: 94.57 % (Epoch - 34) Accuracy on Testset: 87.50 % (Epoch - 35) Accuracy on Trainset: 94.71 % (Epoch - 35) Accuracy on Testset: 87.54 % (Epoch - 36) Accuracy on Trainset: 94.60 % (Epoch - 36) Accuracy on Testset: 85.94 % (Epoch - 37) Accuracy on Trainset: 94.80 % (Epoch - 37) Accuracy on Testset: 86.72 % (Epoch - 38) Accuracy on Trainset: 94.77 % (Epoch - 38) Accuracy on Testset: 87.06 % (Epoch - 39) Accuracy on Trainset: 94.73 % (Epoch - 39) Accuracy on Testset: 87.23 % (Epoch - 40) Accuracy on Trainset: 94.74 % (Epoch - 40) Accuracy on Testset: 88.06 % (Epoch - 41) Accuracy on Trainset: 94.83 % (Epoch - 41) Accuracy on Testset: 87.52 % (Epoch - 42) Accuracy on Trainset: 94.78 % (Epoch - 42) Accuracy on Testset: 87.10 % (Epoch - 43) Accuracy on Trainset: 95.01 % (Epoch - 43) Accuracy on Testset: 87.01 % (Epoch - 44) Accuracy on Trainset: 95.08 % (Epoch - 44) Accuracy on Testset: 86.77 % (Epoch - 45) Accuracy on Trainset: 94.91 % (Epoch - 45) Accuracy on Testset: 87.48 % (Epoch - 46) Accuracy on Trainset: 95.18 % (Epoch - 46) Accuracy on Testset: 86.14 % (Epoch - 47) Accuracy on Trainset: 95.25 % (Epoch - 47) Accuracy on Testset: 88.04 % (Epoch - 48) Accuracy on Trainset: 95.09 % (Epoch - 48) Accuracy on Testset: 86.95 % (Epoch - 49) Accuracy on Trainset: 95.06 % (Epoch - 49) Accuracy on Testset: 87.39 % (Epoch - 50) Accuracy on Trainset: 95.22 % (Epoch - 50) Accuracy on Testset: 87.44 % (Epoch - 51) Accuracy on Trainset: 95.15 % (Epoch - 51) Accuracy on Testset: 87.30 % (Epoch - 52) Accuracy on Trainset: 95.22 % (Epoch - 52) Accuracy on Testset: 84.95 % (Epoch - 53) Accuracy on Trainset: 95.23 % (Epoch - 53) Accuracy on Testset: 87.74 % (Epoch - 54) Accuracy on Trainset: 95.36 % (Epoch - 54) Accuracy on Testset: 87.48 % (Epoch - 55) Accuracy on Trainset: 95.59 % (Epoch - 55) Accuracy on Testset: 87.51 % (Epoch - 56) Accuracy on Trainset: 95.26 % (Epoch - 56) Accuracy on Testset: 87.96 % (Epoch - 57) Accuracy on Trainset: 95.46 % (Epoch - 57) Accuracy on Testset: 87.81 % (Epoch - 58) Accuracy on Trainset: 95.52 % (Epoch - 58) Accuracy on Testset: 87.12 % (Epoch - 59) Accuracy on Trainset: 95.82 % (Epoch - 59) Accuracy on Testset: 86.89 % (Epoch - 60) Accuracy on Trainset: 97.50 % (Epoch - 60) Accuracy on Testset: 89.40 % (Epoch - 61) Accuracy on Trainset: 97.95 % (Epoch - 61) Accuracy on Testset: 89.83 % (Epoch - 62) Accuracy on Trainset: 98.16 % (Epoch - 62) Accuracy on Testset: 89.31 % (Epoch - 63) Accuracy on Trainset: 98.23 % (Epoch - 63) Accuracy on Testset: 89.20 % (Epoch - 64) Accuracy on Trainset: 98.29 % (Epoch - 64) Accuracy on Testset: 89.66 % (Epoch - 65) Accuracy on Trainset: 98.39 % (Epoch - 65) Accuracy on Testset: 89.55 % (Epoch - 66) Accuracy on Trainset: 98.40 % (Epoch - 66) Accuracy on Testset: 89.26 % (Epoch - 67) Accuracy on Trainset: 98.29 % (Epoch - 67) Accuracy on Testset: 88.83 % (Epoch - 68) Accuracy on Trainset: 98.48 % (Epoch - 68) Accuracy on Testset: 89.28 % (Epoch - 69) Accuracy on Trainset: 98.39 % (Epoch - 69) Accuracy on Testset: 89.37 % (Epoch - 70) Accuracy on Trainset: 98.15 % (Epoch - 70) Accuracy on Testset: 88.90 % (Epoch - 71) Accuracy on Trainset: 98.30 % (Epoch - 71) Accuracy on Testset: 88.76 % (Epoch - 72) Accuracy on Trainset: 98.26 % (Epoch - 72) Accuracy on Testset: 89.58 % (Epoch - 73) Accuracy on Trainset: 98.20 % (Epoch - 73) Accuracy on Testset: 88.38 % (Epoch - 74) Accuracy on Trainset: 98.38 % (Epoch - 74) Accuracy on Testset: 89.65 % (Epoch - 75) Accuracy on Trainset: 98.27 % (Epoch - 75) Accuracy on Testset: 88.73 % (Epoch - 76) Accuracy on Trainset: 98.31 % (Epoch - 76) Accuracy on Testset: 89.46 % (Epoch - 77) Accuracy on Trainset: 98.15 % (Epoch - 77) Accuracy on Testset: 89.32 % (Epoch - 78) Accuracy on Trainset: 98.38 % (Epoch - 78) Accuracy on Testset: 89.10 % (Epoch - 79) Accuracy on Trainset: 98.12 % (Epoch - 79) Accuracy on Testset: 88.19 % (Epoch - 80) Accuracy on Trainset: 98.24 % (Epoch - 80) Accuracy on Testset: 88.49 % (Epoch - 81) Accuracy on Trainset: 98.17 % (Epoch - 81) Accuracy on Testset: 89.28 % (Epoch - 82) Accuracy on Trainset: 98.36 % (Epoch - 82) Accuracy on Testset: 88.86 % (Epoch - 83) Accuracy on Trainset: 98.11 % (Epoch - 83) Accuracy on Testset: 88.58 % (Epoch - 84) Accuracy on Trainset: 98.15 % (Epoch - 84) Accuracy on Testset: 89.24 % (Epoch - 85) Accuracy on Trainset: 98.09 % (Epoch - 85) Accuracy on Testset: 89.21 % (Epoch - 86) Accuracy on Trainset: 98.37 % (Epoch - 86) Accuracy on Testset: 88.15 % (Epoch - 87) Accuracy on Trainset: 97.94 % (Epoch - 87) Accuracy on Testset: 88.48 % (Epoch - 88) Accuracy on Trainset: 98.13 % (Epoch - 88) Accuracy on Testset: 88.35 % (Epoch - 89) Accuracy on Trainset: 98.03 % (Epoch - 89) Accuracy on Testset: 88.21 % (Epoch - 90) Accuracy on Trainset: 99.06 % (Epoch - 90) Accuracy on Testset: 90.59 % (Epoch - 91) Accuracy on Trainset: 99.32 % (Epoch - 91) Accuracy on Testset: 90.55 % (Epoch - 92) Accuracy on Trainset: 99.42 % (Epoch - 92) Accuracy on Testset: 90.68 % (Epoch - 93) Accuracy on Trainset: 99.45 % (Epoch - 93) Accuracy on Testset: 90.68 % (Epoch - 94) Accuracy on Trainset: 99.55 % (Epoch - 94) Accuracy on Testset: 90.99 % (Epoch - 95) Accuracy on Trainset: 99.56 % (Epoch - 95) Accuracy on Testset: 90.68 % (Epoch - 96) Accuracy on Trainset: 99.54 % (Epoch - 96) Accuracy on Testset: 90.76 % (Epoch - 97) Accuracy on Trainset: 99.51 % (Epoch - 97) Accuracy on Testset: 90.73 % (Epoch - 98) Accuracy on Trainset: 99.52 % (Epoch - 98) Accuracy on Testset: 90.66 % (Epoch - 99) Accuracy on Trainset: 99.52 % (Epoch - 99) Accuracy on Testset: 90.89 % (Epoch - 100) Accuracy on Trainset: 99.53 % (Epoch - 100) Accuracy on Testset: 90.56 % (Epoch - 101) Accuracy on Trainset: 99.48 % (Epoch - 101) Accuracy on Testset: 90.43 % (Epoch - 102) Accuracy on Trainset: 99.62 % (Epoch - 102) Accuracy on Testset: 90.64 % (Epoch - 103) Accuracy on Trainset: 99.56 % (Epoch - 103) Accuracy on Testset: 90.67 % (Epoch - 104) Accuracy on Trainset: 99.52 % (Epoch - 104) Accuracy on Testset: 90.81 % (Epoch - 105) Accuracy on Trainset: 99.57 % (Epoch - 105) Accuracy on Testset: 91.04 % (Epoch - 106) Accuracy on Trainset: 99.59 % (Epoch - 106) Accuracy on Testset: 90.76 % (Epoch - 107) Accuracy on Trainset: 99.53 % (Epoch - 107) Accuracy on Testset: 90.29 % (Epoch - 108) Accuracy on Trainset: 99.48 % (Epoch - 108) Accuracy on Testset: 90.39 % (Epoch - 109) Accuracy on Trainset: 99.56 % (Epoch - 109) Accuracy on Testset: 90.59 % (Epoch - 110) Accuracy on Trainset: 99.61 % (Epoch - 110) Accuracy on Testset: 90.63 % (Epoch - 111) Accuracy on Trainset: 99.65 % (Epoch - 111) Accuracy on Testset: 90.69 % (Epoch - 112) Accuracy on Trainset: 99.54 % (Epoch - 112) Accuracy on Testset: 90.45 % (Epoch - 113) Accuracy on Trainset: 99.59 % (Epoch - 113) Accuracy on Testset: 91.06 % (Epoch - 114) Accuracy on Trainset: 99.61 % (Epoch - 114) Accuracy on Testset: 90.77 % (Epoch - 115) Accuracy on Trainset: 99.67 % (Epoch - 115) Accuracy on Testset: 90.42 % (Epoch - 116) Accuracy on Trainset: 99.50 % (Epoch - 116) Accuracy on Testset: 90.62 % (Epoch - 117) Accuracy on Trainset: 99.58 % (Epoch - 117) Accuracy on Testset: 90.37 % (Epoch - 118) Accuracy on Trainset: 99.48 % (Epoch - 118) Accuracy on Testset: 89.94 % (Epoch - 119) Accuracy on Trainset: 99.54 % (Epoch - 119) Accuracy on Testset: 90.51 % (Epoch - 120) Accuracy on Trainset: 99.68 % (Epoch - 120) Accuracy on Testset: 91.02 % (Epoch - 121) Accuracy on Trainset: 99.79 % (Epoch - 121) Accuracy on Testset: 91.31 % (Epoch - 122) Accuracy on Trainset: 99.84 % (Epoch - 122) Accuracy on Testset: 91.22 % (Epoch - 123) Accuracy on Trainset: 99.84 % (Epoch - 123) Accuracy on Testset: 91.40 % (Epoch - 124) Accuracy on Trainset: 99.86 % (Epoch - 124) Accuracy on Testset: 91.27 % (Epoch - 125) Accuracy on Trainset: 99.89 % (Epoch - 125) Accuracy on Testset: 91.22 % (Epoch - 126) Accuracy on Trainset: 99.88 % (Epoch - 126) Accuracy on Testset: 91.24 % (Epoch - 127) Accuracy on Trainset: 99.93 % (Epoch - 127) Accuracy on Testset: 91.50 % (Epoch - 128) Accuracy on Trainset: 99.88 % (Epoch - 128) Accuracy on Testset: 90.91 % (Epoch - 129) Accuracy on Trainset: 99.89 % (Epoch - 129) Accuracy on Testset: 91.29 % (Epoch - 130) Accuracy on Trainset: 99.90 % (Epoch - 130) Accuracy on Testset: 91.34 % (Epoch - 131) Accuracy on Trainset: 99.92 % (Epoch - 131) Accuracy on Testset: 91.43 % (Epoch - 132) Accuracy on Trainset: 99.92 % (Epoch - 132) Accuracy on Testset: 91.14 % (Epoch - 133) Accuracy on Trainset: 99.92 % (Epoch - 133) Accuracy on Testset: 91.38 % (Epoch - 134) Accuracy on Trainset: 99.91 % (Epoch - 134) Accuracy on Testset: 91.38 % (Epoch - 135) Accuracy on Trainset: 99.94 % (Epoch - 135) Accuracy on Testset: 91.61 % (Epoch - 136) Accuracy on Trainset: 99.91 % (Epoch - 136) Accuracy on Testset: 91.52 % (Epoch - 137) Accuracy on Trainset: 99.96 % (Epoch - 137) Accuracy on Testset: 91.74 % (Epoch - 138) Accuracy on Trainset: 99.94 % (Epoch - 138) Accuracy on Testset: 91.69 % (Epoch - 139) Accuracy on Trainset: 99.92 % (Epoch - 139) Accuracy on Testset: 91.42 % (Epoch - 140) Accuracy on Trainset: 99.93 % (Epoch - 140) Accuracy on Testset: 91.57 % (Epoch - 141) Accuracy on Trainset: 99.91 % (Epoch - 141) Accuracy on Testset: 91.43 % (Epoch - 142) Accuracy on Trainset: 99.93 % (Epoch - 142) Accuracy on Testset: 91.59 % (Epoch - 143) Accuracy on Trainset: 99.94 % (Epoch - 143) Accuracy on Testset: 91.56 % (Epoch - 144) Accuracy on Trainset: 99.93 % (Epoch - 144) Accuracy on Testset: 91.47 % (Epoch - 145) Accuracy on Trainset: 99.95 % (Epoch - 145) Accuracy on Testset: 91.64 % (Epoch - 146) Accuracy on Trainset: 99.95 % (Epoch - 146) Accuracy on Testset: 91.37 % (Epoch - 147) Accuracy on Trainset: 99.95 % (Epoch - 147) Accuracy on Testset: 91.66 % (Epoch - 148) Accuracy on Trainset: 99.97 % (Epoch - 148) Accuracy on Testset: 91.80 % (Epoch - 149) Accuracy on Trainset: 99.96 % (Epoch - 149) Accuracy on Testset: 91.58 % (Epoch - 150) Accuracy on Trainset: 99.94 % (Epoch - 150) Accuracy on Testset: 91.82 % ``` ::: --- ## 11/26 ### Source code structure #### datasets * cifar10data * cifar-10-batches-py * data_batch_1 ~data_batch_5 * mnist * MNIST * processed * `test.pt` * `training.pt` * raw * t10k-images-idx3-ubyte * t10k-labels-idx1-ubyte * train-images-idx3-ubyte * train-labels-idx1-ubyte --- #### ResNet * inputmap of ResNet (it's PT file) * `main.py` : main program, parser is define here * `models_resnet.py` : define the model of ResNet in this work, here we make ResNet18 for testing * `models_resnet_inputmap.py` * `models_vgg.py` : define the model of VGG in this work * `models_vgg_inputmap.py` * `utils.py` : define several funcs use in this work * `run.sh` : run all file, just like makefile --- #### VGG * inputmap of VGG * other files are same with files in ResNet --- #### venv * environment data ---- #### Code tracing ##### `main.py` [source](https://hackmd.io/@hsieh22/fault_tolerant_rram_src1) * define and implement parser * parser will define parameter how to control program, then it will reference parameters to run training, finetune and evaluation * we can set parser, resume (directory that storing weight),the bits of quantizing, network architecture or restoring data * define quantize_opt * define LeNet * deal with different datasets * inference step * finetune step * quantize options * training ---- ##### `utils.py` * implement all details of function * Variation() * DynamicVariation() * quantize() * deal with three type of quantize : linear, exp and power * SaveWeights() * RowRestoring_Weights() * RowRestoring_Gradients() * Train() * Eval() --- ### TODO :::info 1. realize the structure of this work 2. trace each process in `main.py` 3. tracing `utils.py` :::