BDSE 深度學習

[TOC] # 課程資訊 * [[講義下載]]( https://www.dropbox.com/s/673wyuwqk5a37rx/BDSE_v2023.07.pdf?dl=1) * [[資料集下載]]( https://www.dropbox.com/s/vn1efjse01qvhrn/BDSE_v2023.03.tar?dl=1) * [[課程補充筆記本]](https://share.goodnotes.com/s/VmX1d95YwrPNJGffvSeDpZ) # 作業 1. Q: 在**模型權重數量差不多**時, 我們有兩種建模策略: **(A) 寬但淺** (淺層網路; 隱藏層很少, 中間只激活少數次; 網路層內有較多神經元) **(B) 窄但深** (深層網路; 中間會激活很多次; 網路層內有較少神經元) 這兩種建模策略哪一個可能比較好呢? 你可能要建立一個統計表格, 該表格包含: `model_setting`, `num_weights`, `accuracy`等欄位. 這樣會比較好回答此申論問題．不用太嚴謹或複雜. 簡單實驗, 給出一些說明即可． 2. 使用 [learning rate排程器](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler) 將MLP模型的訓練, 做學習率排程. # 7/22 ## 課綱 - [x] CNN Continue - [x] History (LeNet, VGGNet, GoogleNet, ResNet, DenseNet) - [x] Code Demo - [x] MNIST - [ ] ~~(How to build ResNet)~~ (ResNet building block implementation as an exercise) - [x] More on Optimizers - [x] Vanilla RNN - [x] LSTM # 7/11 ## 課綱 - [x] Code Refactoring - [x] TF Keras Model Construction API: Create a custom layer - [x] BatchNorm - [x] Backpropagation - [x] Activation - [ ] Convolutional Neural Network - [x] Basic theories - [ ] Model overview - [ ] Know how to build ResNet ## 補充 * Custom Layer ```python= class MyLayer(tf.keras.layers.Layer): def __init__( self, num_inputs: int = 3, num_units: int = 5, num_filters: int = 5, activation: str = "relu", **kwargs, ) -> None: super().__init__(**kwargs) self.dense1 = tf.keras.layers.Dense( num_units, input_shape=(num_inputs,), activation=activation, ) self.dense2 = tf.keras.layers.Dense( num_units, activation=activation, ) self.conv1 = tf.keras.layers.Conv1D( num_filters, 3, padding='same', input_shape=(num_inputs, 1), activation=activation, ) self.conv2 = tf.keras.layers.Conv1D( num_filters, 3, padding='same', activation=activation, ) self.flatten = tf.keras.layers.Flatten() def __call__(self, x): x1 = self.dense1(x) # [BS, num_inputs] -> [BS, num_units] x1 = self.dense2(x1) # [BS, num_units] -> [BS, num_units] x2 = tf.expand_dims(x, axis=-1) # [BS, num_inputs] -> # [BS, num_inputs, 1] x2 = self.conv1(x2) # [BS, num_inputs, 1] -> [BS, num_inputs, num_filters] x2 = self.conv2(x2) # [BS, num_inputs, num_filters] -> [BS, num_inputs, num_filters] x2 = self.flatten(x2) # [BS, num_inputs, num_filters] -> [BS, num_inputs*num_filters] print(f"x1 shape={x1.shape}") print(f"x2 shape={x2.shape}") return tf.concat([x1, x2], -1) # x1 output: [BS, num_units] # x2 output: [BS, num_inputs*num_filters] # (1) use "concat" to combine: # final output shape: [BS, num_units + num_inputs * num_filters] # (2) use "+" to combine: # final output shape: [BS, num_units] = [BS, num_inputs*num_filters] # (however, the condition `num_units=num_inputs*num_filters` must hold) ``` * Tensor shape變換 ```python= import numpy as np tmp = np.arange(5) tmp.reshape((5, 1)).squeeze() tmp.reshape((5, -1)).shape tmp[..., np.newaxis].shape tmp[...].shape tmp[..., np.newaxis].shape tmp[np.newaxis, ...].shape np.expand_dims(tmp, axis=-1).shape np.expand_dims(tmp, axis=0).shape [np.expand_dims(tmp, axis=0)]*2 [array([[0, 1, 2, 3, 4]]), array([[0, 1, 2, 3, 4]])] np.concatenate([np.expand_dims(tmp, axis=0)]*2, axis=0).shape np.concatenate([np.expand_dims(tmp, axis=0)]*3, axis=0).shape ``` * Some layer's input / output shape * **Dens**: `[num_samples=batch_size, num_features]` -> `[num_samples, num_dense_units]` * **Conv2D**: `[num_samples=batch_size, Height, Width, num_channels]=[N, H, W, C]` -> `[num_samples, feature_map_height, feature_map_width, num_filters]` * **Conv1D**: `[num_samples=batch_size, Width, num_channels]` -> `[num_samples, feature_map_width, num_filters]` * **RNN / LSTM**: `[num_samples=batch_size, num_time_steps, num_features]` * Subclassing-API * 簡單兩層Dense堆疊的網路層 ```python= class MyLayer(tf.keras.layers.Layer): def __init__(*args, **kwargs): super().__init__(*args, **kwargs) self.dense1 = tf.keras.layers.Dense(...) self.dense2 = tf.keras.layers.Dense(...) def __call__(x, ): # 定義正向傳遞邏輯 return self.dense2(self.dense1(x)) ``` 等效於: ```python= model = tf.keras.models.Sequential() model.add(tf.keras.layers.Dense(...)) model.add(tf.keras.layers.Dense(...)) ``` * 兩個Conv, 兩個Dense組成的網路層 * Conv1 * [Pydantic](https://docs.pydantic.dev/latest/) * [Pydantic-argparse](https://pydantic-argparse.supimdos.com/usage/arguments/) * turn on inlay type hints in vscode ![](https://hackmd.io/_uploads/SJEkUN5Fn.png) # 7/9補充 * 深層網路為何需要activation function? * $(\Gamma_{-20^\circ})(\Gamma_{+30^\circ})\vec{x} = \Gamma_{+10^\circ} \vec{x}$ * $(\Gamma_{-20^\circ})(K)(\Gamma_{+30^\circ})\vec{x} \neq \Gamma_{+10^\circ} \vec{x}$ * 矩陣K在這裡類比於一種擾動。該擾動使得**兩個**用於旋轉某向量的**旋轉矩陣**，無法**等效於一個旋轉矩陣**。我們用這樣的例子來表示，深層神經網路，激活函數(activation function)存在的必要性。Activation function在上述例子，相當於用做於擾動的矩陣K。 * Model finetuning(調參)時，若不用多折交叉驗證 (太耗時)，則可考慮將資料切三份，以驗證模型表現: TRAIN+VAL用做 **模型選擇與訓練** "**Model Training + Model Selection**"; TEST用做 **模型效能驗證**。![](https://i.imgur.com/69CPUHp.png =600x) * Recall或Precision的理解方式 ![](https://hackmd.io/_uploads/rymIz0PYn.jpg =600x) * Loss為MSE時, Maximum Likelihood 等同於 Minimize Loss ![](https://hackmd.io/_uploads/rJZwacDFh.jpg =600x) * Gradient Descent 公式 ![](https://hackmd.io/_uploads/SykbohDt3.jpg =600x) * Stochastic Gradient Descent; 以及何謂Epoch, 何謂Iteration ![](https://hackmd.io/_uploads/SJ1qshvYh.jpg =600x) * Cross Entropy Loss如何得到 (從統計誤差分佈假設出發)![](https://hackmd.io/_uploads/HyZxahvYn.jpg =600x) * Softmax網路層如何運作 ![](https://hackmd.io/_uploads/SkmgpnDKn.jpg =600x) # 7/8 補充 * 補充: Docker image 下載位置 * [DockerHub Official TensorFlow](https://hub.docker.com/r/tensorflow/tensorflow/tags) * [NGC TensorFlow Image](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) * [NGC TensorFlow Release Notes]( https://docs.nvidia.com/deeplearning/frameworks/tensorflow-release-notes/) # 以Poetry 安裝環境假設已經有: * Anaconda Python * NVIDIA GPU Driver * CUDA * cuDNN --- 1. 開啟anaconda prompt, 建立conda開發環境`(dlenv)` ```bash conda create -n dlenv python==3.10.9 ipython==8.10.0 ``` 上述指令執行後, 按yes確認, 即可完成環境`(dlenv)`的建立. :::info :bulb: 你必須在conda `(base)` 的終端機環境執行上述指令. 如果你不是, 請透過`conda deactivate`離開某conda環境, 以回到conda的`(base)`環境. ::: 2. 切換至conda開發環境 `(dlenv)` ```bash conda activate dlenv ``` 3. 安裝Poetry ~~1. 於powershell一鍵安裝 Poetry~~ ~~2. 將`C:\Users\student\AppData\Roaming\Python\Scripts`加至PATH~~ :::info :bulb:之前我們嘗試在系統端安裝poetry.這可能不太好. 因為我們的DL開發環境`(dlenv)`, 是依附在conda內, 所以我們可能必須改在DL開發環境`(dlenv)`內安裝poetry (`pip install poetry`). 在DL開發環境`(dlenv)`內安裝poetry, 我們就可以避免poetry控制其他位置的python. 也就是說, 我們可縮限poetry只能管理DL開發環境`(dlenv)`內的套件. 如果你先前誤執行了: ```bash= (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py - ``` 請透過以下指令將系統內的poetry移除: ```bash= (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py - --uninstall ``` ::: 請至`(dlenv)`環境, 執行`pip install poetry`以安裝套件管理工具. 3. 以vscode進入課程資料夾 `C:\Users\student\Desktop\BDSE_v2023.07\BDSE_v2023.03` 1. 開啟課程資料夾內的 `pyproject.toml` 檔案, 並於`[tool.poetry.dependencies]`段落, 編輯TensorFlow相關的套件: ```bash tensorflow = "2.10.1" tensorflow-addons = "^0.19.0" tensorflow-io-gcs-filesystem = "0.31.0" ``` 2. 終端機進入DL開發環境`(dlenv)`後, 執行`poetry lock` (可以確立套件相依關係. 並將環境所有套件版本固定. 這個lock以及`pyproject.toml`文件之後可以給別人, 如此, 別人就會跟你是裝同樣的套件.) 3. 確定終端機在DL開發環境`(dlenv)`後, 執行`poetry install` 4. 安裝一些vscode延伸套件: * pylance, python * toml * reload * rainbow csv * Black Formatter (Microsoft) * 設定 -> 搜尋formatting -> formatting Provider 選 Black ![](https://hackmd.io/_uploads/ryw0EYUt3.png) * 搜尋on save -> Editor: Format on save確定打勾 * isort (Microsoft) * 需要進去settings.json追加一些設定才能format on save.設定如下: ```json { "editor.fontSize": 18, "liveServer.settings.donotVerifyTags": true, "liveServer.settings.donotShowInfoMsg": true, "editor.wordWrap": "on", "editor.formatOnSave": true, "[python]": { "editor.defaultFormatter": "ms-python.black-formatter", "editor.formatOnType": true, "editor.formatOnSave": true, "editor.codeActionsOnSave": { "source.organizeImports": true }, }, "isort.args": [ "--profile", "black", ], "terminal.integrated.defaultProfile.windows": "Command Prompt", "python.formatting.provider": "black", "python.defaultInterpreterPath": "C:\\Users\\student\\anaconda3\\python.exe", "window.zoomLevel": -1, "notebook.formatOnSave.enabled": true, } ``` * Remote SSH (Microsoft) * Concept: * Linting: Pylance * Formatting: Black * coding style * google style: https://google.github.io/styleguide/pyguide.html * to allow people be happy about your functions, classes: * type annotation is important * docstring is also important --- 改用 pip 安裝 https://chat.openai.com/share/cc46158f-4844-4b0a-9dd9-f7b0e6cd2f98 # GPU環境設置可能的安裝方式: 1. Windows -> NVIDIA GPU Driver -> WSL2(Linux) -> Docker -> 啟動TensorFlow, PyTorch等官方維護的 Docker image * Google TensorFlow(+keras) / Facebook PyTorch Docker image 裡面暗藏了 CUDA (NVIDIA GPU開發框架), cuDNN(Deep Neural Network 函式庫), TensorFlow 2. Linux -> NVIDIA GPU Driver -> Docker -> 啟動TensorFlow, PyTorch等官方維護的 Docker image 3. Windows -> NVIDIA Driver (NVIDIA顯卡驅動) → CUDA (NVIDIA顯卡軟體開發框架) → cuDNN (用於NVIDIA顯卡運算的神經網路函式庫) → TensorFlow。 ## GPU相關套件 - 於 Windows 內直接安裝安裝順序: NVIDIA Driver (NVIDIA顯卡驅動) → CUDA (NVIDIA顯卡軟體開發框架) → cuDNN (用於NVIDIA顯卡運算的神經網路函式庫) → TensorFlow。 0. 安裝[NVIDIA Driver](https://www.nvidia.com/Download/index.aspx) (此步驟可略過，因為教室的機器應該已經安裝好NVIDIA驅動程式) 1. 啟動 `NVSMI` (NVIDIA System Management Interface) 開啟終端機 (如GIT BASH或Windows命令提示字元), 輸入`nvidia-smi`後, 按Enter, 即可顯示出GPU使用率等資訊。此步驟若能成功執行，也代表驅動程式應已正常安裝。 2. 安裝[CUDA v11.6.1](https://developer.nvidia.com/cuda-toolkit)。[[下載連結]](https://developer.download.nvidia.com/compute/cuda/11.6.1/local_installers/cuda_11.6.1_511.65_windows.exe) 3. 安裝[cuDNN v 8.4.0 (需與CUDA v11.6.1相容)](https://developer.nvidia.com/cudnn)。[[下載連結]](https://www.dropbox.com/s/vg4yp1cf5hvf2p3/cudnn-windows-x86_64-8.4.0.27_cuda11.6-archive.zip?dl=1) 將解壓縮後的cuDNN資料夾內的檔案, 逐一複製到```C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6```。 4. 執行以下指令，確定NVIDIA CUDA Compiler (NVCC)有被安裝: ```nvcc -V``` 如果有正常執行，表示CUDA應該可以使用。 5. 安裝TensorFlow: 1. 執行 ```pip install tensorflow-gpu``` 即可完成安裝。 2. 檢查TensorFlow是否可把張量丟到GPU ```python In [1]: import tensorflow as tf In [2]: tf.__version__ Out[2]: '2.10.1' In [3]: import numpy as np In [4]: tmp = np.random.normal(0, 1, (3, 3)) In [5]: tmp Out[5]: array([[-0.58497332, 0.91560168, 0.35745953], [ 0.36502097, 0.81025645, 1.17180727], [ 0.50818861, -1.05783736, -1.82408266]]) In [6]: tmp = tmp.astype(np.float32) In [7]: tmp.dtype Out[7]: dtype('float32') In [8]: tmp = tf.constant(tmp) 2021-12-28 20:23:09.054421: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-12-28 20:23:09.522473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2774 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1 In [9]: tmp.device Out[9]: '/job:localhost/replica:0/task:0/device:GPU:0' ``` 如有出現`GPU:0`字眼，表示GPU已可正常使用。 6. 修復TensorFlow Bug 很不幸的，目前(2022年12月)，發現新版TensorFlow `v2.10.1`在Windows系統上使用卷基層做模型推論的時候會遇到錯誤 (與`zlibwapi.dll`相關)。如果你有遇到這個問題，可將: `C:\Program Files\NVIDIA Corporation\Nsight Systems 2021.5.2\host-windows-x64\zlib.dll` 更名，並且複製到 `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\zlibwapi.dll` 。複製完後，簡單將一個隨機張量丟進一個卷積層，即可確認問題是否解決。 # 參考連結 * https://www.youtube.com/@ArjanCodes * Find state-of-art model implementations (if papers are with codes :arrow_right: :100:): https://paperswithcode.com * Understand CNN basics (:100: you must read this, understand every lines of words): https://cs231n.github.io * karparthy: https://www.youtube.com/results?search_query=karparthy * Learn Deep Learning basics (Wow! Learning by doing! :100:): https://zh.d2l.ai * Machine Learning basic theories (:100: for those who wants to understand every general bits of maths in Machine Learning): https://cs229.stanford.edu/notes2022fall/main_notes.pdf * Understand Backpropagation: https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf * Python Machine Learning (Sebastian Raschka) * Keras book (API參考用?) https://www.manning.com/books/deep-learning-with-python --- wengchihung@gmail.com