BDSE32 - 深度學習

# BDSE32 - 深度學習 # 講義與程式範例下載 * [講義](https://www.dropbox.com/scl/fi/zfd8xpi040j5ll8afjp7y/BDSE_v2023.11.pdf?rlkey=3z8wjoxptbpl0ehn66il20zmu&dl=1) * [程式範例](https://www.dropbox.com/scl/fi/tunvxaaazozj7781euw18/BDSE_DL_v2023.11.tar.gz?rlkey=2n9xxbme2mzh3odskz9wgvhsv&dl=1) * [補充筆記](https://share.goodnotes.com/s/gYPxKKP7gle2TvlcRmXZnv) # To Do #### 2023/11/04 - [x] How to prepare Deep-learning DEV environment - [x] Connection between Loss Minimization and Likelihood Maximization - [x] What is Stochastic Gradient Descent - [x] common terms: batch, epoch, iteration #### 2023/11/05 - [x] How to build a Linear Regressor - [x] Understand Loss: Mean Square Error, Cross Entropy, L1 regularization - [x] What is recall, precision? - [x] Deep-learning applications: from past up to now - [x] Understand Backpropagation - [x] Start getting used to tf.keras - [x] Get to know CNN #### 2023/11/09 - [ ] APIs for model construction - [x] `tf.keras.models.Sequential` - [x] `tf.keras.models.Model` - [x] `tf.keras.layers.Layer` - [x] Introduction to CNN models # 2023/11/23上課補充 ## 煉丹 - [Bag of tricks](https://arxiv.org/pdf/1812.01187.pdf) - [Bag of freebies (object detection)](https://arxiv.org/abs/2207.02696) ## 三種拿到accuracy的方法 ```python= # 練習於此 pred_y = model.predict(x_test).argmax(axis=1) # Shape=[num_samples, num_classes] -> [num_samples,] # 三種拿到accuracy的方法 print("accuracy=", (pred_y == y_test).sum() / len(y_test)) print("accuracy=", (pred_y == y_test).mean()) model.evaluate(x_test, y_test_one_hot) ``` ## 固定特徵擷取器 1. 特徵擷取器可學習: ```(CONV(trainable=True) ->CONV(trainable=True) ->CONV(trainable=True) -> ... ->CONV(trainable=True) ->CONV(trainable=True) ->) -> Flatten -> Dense``` 2. 固定特徵擷取器 (特徵擷取器不可學習): ```(CONV(trainable=False) ->CONV(trainable=False) ->CONV(trainable=False) -> ... ->CONV(trainable=False) ->CONV(trainable=False) ->) -> Flatten -> Dense``` * 應用情境:數據不夠多, 不想要把人家訓練的強特徵擷取器洗掉. 3. 固定特徵擷取器 (特徵擷取器部分可學習): ```(CONV(trainable=False) ->CONV(trainable=False) ->CONV(trainable=False) -> ... ->CONV(trainable=True) ->CONV(trainable=True) ->) -> Flatten -> Dense``` * 應用情境:數據不夠多, 想要微調模型而已. * 利用少量數據高學習率訓練的話很有可能會把很厲害的特徵擷取器弄壞掉. 用低學習率訓練, 微調特徵擷取器的末端的幾個conv層. ## ResBlock ```python= class Residual(tf.keras.layers.Layer): """Class for the building block of ResNet V2. Reference: "Identity Mappings in Deep Residual Networks"; https://arxiv.org/abs/1603.05027 Code in principle follows: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/model_zoo/vision/resnet.py """ def __init__( self, filters, downsample=False, strides=1, use_bias=True, epsilon=1e-3, axis=-1, l2_strength=1.0e-5, **kwargs ): """Initialize all the necessary layers and parameters for the `Residual` block. Args: filters: Number of filters of all the used Conv layers. downsample: If to reduce size and/or channels of the input feature map. strides: Strides of the first Conv layers. If >1, the input feature map will be down-sampled. use_bias: If `True`, all the Conv layers will have trainable biases. """ super(Residual, self).__init__(**kwargs) if axis == -1 or axis == 3: data_format = "channels_last" elif axis == 1 or axis == -3: data_format = "channels_firt" else: raise ValueError("Data format invalid.") self.conv1 = tf.keras.layers.Conv2D( filters, kernel_size=3, strides=strides, use_bias=use_bias, data_format=data_format, kernel_regularizer=tf.keras.regularizers.l2(l2_strength), padding="SAME", ) self.conv2 = tf.keras.layers.Conv2D( filters, kernel_size=3, strides=1, use_bias=use_bias, data_format=data_format, kernel_regularizer=tf.keras.regularizers.l2(l2_strength), padding="SAME", ) self.downsample = downsample if self.downsample: self.conv_down = tf.keras.layers.Conv2D( filters, kernel_size=1, strides=strides, use_bias=use_bias, data_format=data_format, kernel_regularizer=tf.keras.regularizers.l2(l2_strength), padding="SAME", ) norm_params = {"epsilon": epsilon, "axis": axis} self.norm1 = tf.keras.layers.BatchNormalization(**norm_params) self.norm2 = tf.keras.layers.BatchNormalization(**norm_params) def call(self, x, training=None): # 實作練習 # self.norm1, self.norm2, self.conv1, self.conv2, self.conv_down # x = self.norm1(x, training=training) residual = x x = self.conv1( tf.nn.relu(x), ) x = self.conv2( tf.nn.relu( self.norm2(x, training=training), ), ) if self.downsample: residual = self.conv_down(residual) return x + residual ``` ## DenseBlock ```python= class DenseBlock(tf.keras.layers.Layer): """Class for the building block of ResNet V2. Reference: "Identity Mappings in Deep Residual Networks"; https://arxiv.org/abs/1603.05027 Code in principle follows: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/model_zoo/vision/resnet.py """ def __init__( self, filters, downsample=False, strides=1, use_bias=True, epsilon=1e-3, axis=-1, l2_strength=1.0e-5, **kwargs ): """Initialize all the necessary layers and parameters for the `Residual` block. Args: filters: Number of filters of all the used Conv layers. downsample: If to reduce size and/or channels of the input feature map. strides: Strides of the first Conv layers. If >1, the input feature map will be down-sampled. use_bias: If `True`, all the Conv layers will have trainable biases. """ super(DenseBlock, self).__init__(**kwargs) if axis == -1 or axis == 3: data_format = "channels_last" elif axis == 1 or axis == -3: data_format = "channels_firt" else: raise ValueError("Data format invalid.") self.conv1 = tf.keras.layers.Conv2D( filters, kernel_size=3, strides=strides, use_bias=use_bias, data_format=data_format, kernel_regularizer=tf.keras.regularizers.l2(l2_strength), padding="SAME", ) self.conv2 = tf.keras.layers.Conv2D( filters, kernel_size=3, strides=1, use_bias=use_bias, data_format=data_format, kernel_regularizer=tf.keras.regularizers.l2(l2_strength), padding="SAME", ) self.downsample = downsample if self.downsample: self.conv_down = tf.keras.layers.Conv2D( filters, kernel_size=1, strides=strides, use_bias=use_bias, data_format=data_format, kernel_regularizer=tf.keras.regularizers.l2(l2_strength), padding="SAME", ) norm_params = {"epsilon": epsilon, "axis": axis} self.norm1 = tf.keras.layers.BatchNormalization(**norm_params) self.norm2 = tf.keras.layers.BatchNormalization(**norm_params) self.concat = tf.keras.layers.Concatenate(axis=axis) def call(self, x, training=None): # 實作練習 # self.norm1, self.norm2, self.conv1, self.conv2, self.conv_down # x = self.norm1(x, training=training) residual = x x = self.conv1( tf.nn.relu(x), ) x = self.conv2( tf.nn.relu( self.norm2(x, training=training), ), ) if self.downsample: residual = self.conv_down(residual) return self.concat([x, residual]) ``` # 2023/11/04上課補充 #### 清理`._`開頭的垃圾檔案 ```python from pathlib import Path if __name__ == "__main__": to_clean_dir = Path("C:\\Users\\student\\Desktop\\BDSE_DL_v2023.11.tar\\BDSE_DL_v2023.11\\BDSE_DL_v2023.11") for path in to_clean_dir.glob("**/._*"): path.unlink() ``` #### 型別簡介 * `float32` (single precision) -> 表示採用(32/8=) 4 bytes儲存單一浮點數 * `float64` (double precision) -> 表示採用(64/8=) 8 bytes儲存單一浮點數 * `uint8` (使用1 byte儲存單一無負號整數) * 舉例: 一張800x600的彩色圖片若無壓縮(每個像素以uint8型別儲存)會占用記憶體 800*600*3*1 / 1024 / 1024 = 1.37...(Mb) # 2023/11/05上課補充 #### DataLoader ```python= from collections.abc import Sequence import typing as t import typing_extensions as tx import numpy.typing as npt Float32Or64Array: t.TypeAlias = t.Union[npt.NDArray[np.float64], npt.NDArray[np.float32]] class DataLoader(Sequence): """此方法負責批次性的拋出資料給機器學習的演算法做訓練。""" def __init__( self, x: Float32Or64Array, y: Float32Or64Array, batch_size: int, shuffle: bool = True, ) -> None: self.x = x # [num_samples, num_features] self.y = y # [num_samples,] self.bs = batch_size # 每批資料的大小。 self.shuffle = shuffle assert len(self.x) == len(self.y) self.indices = np.arange(len(self)) # len(self)相當於 self.__len__() # 若bs=128, 資料量=512, 則self.indices = [0, 1, 2, 3] @tx.override def __iter__(self): while True: if self.shuffle: np.random.shuffle(self.indices) for j in self.indices: yield self[j] # 相當於self.__getitem__(j) def __len__(self) -> int: return len(self.x) // self.bs # number of batches def __getitem__(self, j: int) -> tuple[npt.NDArray[np.float32], npt.NDArray[np.float32]]: x_batch = self.x[ self.bs * j : self.bs * (j + 1), : ] # [num_samples, num_features] -> [batch_size, num_features] y_batch = self.y[self.bs * j : self.bs * (j + 1)] # [num_samples,] -> [batch_size,] return x_batch.astype(np.float32), y_batch.astype(np.float32) ``` #### LinearRegressor ```python= class LinearRegressor: """此類別處理線性回歸 (+L1正規項)。""" def __init__(self, num_features: int) -> None: # 將權重初始化 self.W = tf.Variable(tf.random.normal((num_features, 1), dtype=np.float32)) # [num_features, 1] self.b = tf.Variable(tf.random.normal((1,), dtype=np.float32)) # [1,] def __call__(self, x: tf.Tensor): """輸入x，利用當前的權重參數(W和b)來預測y。""" y_pred = tf.matmul(x, self.W) + self.b # [BS, 1] # x: Shape=[BS, num_features] # self.W: Shape=[num_features, 1] # x * self.W: Shape=[BS, 1] # b: Shape=[1,] ->broadcast-> [BS, 1] # x * self.W + b: Shape=[BS, 1] return tf.squeeze(y_pred) # [BS,] def _mean_square_error_loss(self, y_true: tf.Tensor, y_pred: tf.Tensor) -> tf.Tensor: """此為Mean Square Error Loss。""" # y_true: Shape=[BS,] # y_pred: Shape=[BS,] return tf.reduce_mean((y_true - y_pred) ** 2, keepdims=False) # [BS,] def _sgd_update_weights(self, grad_W: tf.Variable, grad_b: tf.Variable, lr: float) -> None: """使用Mini-batch Gradient Descent來做權重更新。""" self.W.assign_sub(lr * grad_W, name="update_W") # W := W- lr * grad_W self.b.assign_sub(lr * grad_b, name="update_b") # b := b- lr * grad_b def fit( self, x: Float32Or64Array, y: Float32Or64Array, batch_size: int, C: float, num_epochs: int, lr: float = 0.001, info_step: int = 1, verbose: bool = True, ) -> tf.Tensor: """以批次更新方式，尋找最適回歸權重。""" # 取得用於產生批次資料的generator data_generator = DataLoader(x, y, batch_size) # 開始最小化Loss (藉由更新 W, b) for epoch in range(num_epochs): for idx, (x, y_true) in zip(range(len(data_generator)), data_generator): with tf.GradientTape() as tape: # 1. x 輸入模型，正向傳遞後，變成y_pred # 2. y_pred和y_true做比較後，得到loss y_pred = self(x) # 　self.__call__(x) loss = self._mean_square_error_loss(y_true, y_pred) loss += (1.0 / C) * tf.reduce_sum(tf.abs(self.W)) # 追加L1懲罰項 grad_W, grad_b = tape.gradient(loss, [self.W, self.b]) self._sgd_update_weights(grad_W, grad_b, lr) # 使用梯度下降法更新 W, b if verbose and epoch % info_step == 0: loss_eval = self._mean_square_error_loss(y_true, y_pred) print("epoch %3i," % epoch, "loss=", loss_eval.numpy()) return self._mean_square_error_loss(y_true, y_pred) # 輸出訓練後的模型Loss ``` #### L1規範項/懲罰項 (L1 regularization/ L1 loss penalty) $\tilde{L} = L(w_0, w_1, w_2, ..., w_N) + \frac{1}{C} L_1(w_0,w_1, w_2, ..., w_N)$, $L_1(w_0, w_1, w_2, ..., w_N) = \sum_j |w_j|$. 其中, `C`為"懲罰項強度"的倒數. * 若$L=0$ (只存在L1規範項), 目標則僅為找到恰當的$w_0, w_1, w_2, ..., w_N$使得$L1= \sum_j |w_j|$最小. * 此答案很簡單: 已知$w_0=w_1=w_2=...=w_N=0$時, $L_1$為最小($L_1=0$). * 可見L1項的存在有機會壓低模型權重的數值大小. * 由於模型為$\hat{y}=w_0 + w_1 x_1 + w_2 x_2 + ... + w_N x_N$. 若 $w_2$被降低至0, 表示與之相乘的特徵欄位($x_2$)可能不是有用特徵 (所以模型才學習$w_2=0$). * 我們說L1可鼓勵: * 權重矩陣稀疏. * 換句話說, L1可鼓勵模型忽略無用特徵, 不要將無用特徵納入模型預測. * 換句話說, L1可用於特徵選取(feature selection). #### MLP練習 ![截圖 2023-11-05 下午2.27.49.png](https://hackmd.io/_uploads/Sk6UL3E7p.png) ```python= from keras.layers import Dense from keras.losses import CategoricalCrossentropy from keras.optimizers import SGD from keras.models import Sequential import matplotlib.pyplot as plt import numpy as np from keras.utils import to_categorical X = np.zeros((90, 3)) for j in range(0, 30): X[j, :] = 0.0 for j in range(30, 60): X[j, :] = 1.0 for j in range(60, 90): X[j, :] = 2.0 y = X[:, 0] y_new = to_categorical(y) # 準備開始訓練 lr = 0.05 # 設定學習率 num_epochs = 1000 # 設定優化次數 num_units = 1024 act = 'relu' # Build Model layers = [ Dense(num_units, activation=act, input_shape=(3,)), Dense(num_units, activation=act), Dense(num_units, activation=act), Dense(num_units, activation=act), Dense(num_units, activation=act), Dense(num_units, activation=act), Dense(num_units, activation=act), Dense(3, activation='softmax'), ] model = Sequential() for layer in layers: model.add(layer) # model.summary() # Compile Model model.compile( loss=CategoricalCrossentropy(), optimizer=SGD(learning_rate=lr), metrics=['accuracy'], ) # Fit Model hist = model.fit(x=X, y=y_new, epochs=num_epochs) # 畫出訓練過程 plt.plot(hist.history['accuracy'], ms=5, marker='o', label='accuracy') plt.legend() plt.show() plt.plot(hist.history['loss'], ms=5, marker='o', label='loss') plt.legend() plt.show() ``` # 2023/11/09上課補充 #### MLP練習模型重構, 並追加`學習率排程器`和`最好模型儲存器` ```python= from keras.layers import Dense from keras.losses import CategoricalCrossentropy from keras.optimizers import SGD from keras.models import Sequential import matplotlib.pyplot as plt import numpy as np from keras.utils import to_categorical from keras.callbacks import LearningRateScheduler, ModelCheckpoint import tensorflow as tf from collections import defaultdict def scheduler(epoch, lr): if epoch < 200: return lr else: return lr * tf.math.exp(-0.005) lr = 0.1 out = defaultdict(list) for epoch in range(1000): lr = scheduler(epoch, lr) out['lr'].append(lr) out['epoch'].append(epoch) plt.plot(out['epoch'], out['lr']) plt.show() scheduler_callback = LearningRateScheduler(schedule=scheduler) model_checkpointer_callback = ModelCheckpoint( "./model_checkpoints/", monitor='loss', save_best_only=True, initial_value_threshold=0.0005, ) X = np.zeros((90, 3)) for j in range(0, 30): X[j, :] = 0.0 for j in range(30, 60): X[j, :] = 1.0 for j in range(60, 90): X[j, :] = 2.0 y = X[:, 0] y_new = to_categorical(y) # 準備開始訓練 # lr = 0.05 # 設定學習率 lr = 0.3 # 設定學習率 num_epochs = 1000 # 設定優化次數 num_units = 1024 act = 'relu' num_hidden_layers = 6 # Build Model layers: list[Dense] = [ Dense(num_units, activation=act, input_shape=(3,)), *[ Dense( num_units, activation=act, name=f"hidden_dense_{j}", ) for j in range(num_hidden_layers) ], Dense(3, activation='softmax'), ] model = Sequential() for layer in layers: model.add(layer) # model.summary() # Compile Model model.compile( loss=CategoricalCrossentropy(), optimizer=SGD(learning_rate=lr), metrics=['accuracy'], ) # Fit Model hist = model.fit( x=X, y=y_new, epochs=num_epochs, callbacks=[ scheduler_callback, model_checkpointer_callback, ], ) # 畫出訓練過程 plt.plot(hist.history['accuracy'], ms=5, marker='o', label='accuracy') plt.legend() plt.show() plt.plot(hist.history['loss'], ms=5, marker='o', label='loss') plt.legend() plt.show() ``` 進階排程器設定可參考: https://arxiv.org/pdf/1812.01187.pdf ### 建模API #### `tf.keras.models.Sequential` ```python= model = tf.keras.models.Sequential() model.add(my_first_layer) model.add(my_second_layer) ... ``` 流程: x -> my_first_layer -> my_second_layer -> out --- #### `tf.keras.models.Model` ```python= out1 = layer1(in_tensor) out2 = layer2(in_tensor) out_tensor = concat([out1, out2]) model = tf.keras.models.Model(in_tensor, out_tensor) ``` 流程: 1. x -> copy(x) -> layer1 -> out1 2. x -> copy(x) -> layer2 -> out2 3. out = concat([out1, out2]) #### `tf.keras.layers.Layer` ```python= class MyLayer(tf.keras.layers.Layer): def __init__(self, **kwargs): super().__init__(**kwargs) self.layer1 = Layer1Constructor(...) self.layer2 = Layer2Constructor(...) def call(x, training=True): out1 = self.layer1(x, training=training) out2 = self.layer2(x, training=training) return concat([out1, out2]) ``` 流程: 1. x -> copy(x) -> layer1 -> out1 2. x -> copy(x) -> layer2 -> out2 3. out = concat([out1, out2]) # Python Coding Style * Linter: `pylance` * Formatter: `black`, `isort` * 設置Black (format on save) ![未命名.png](https://hackmd.io/_uploads/S17227Q7a.png) * 設置isort (sort on save) 將下述設定加到`settings.json`: ```json "editor.codeActionsOnSave": { "source.organizeImports": true } ``` ![未命名2.png](https://hackmd.io/_uploads/rkOLAmX7p.png) * 加強自己程式碼的易讀性 * 善用`docstring`, `type hint` * follow, e.g., [Google's Python Coding Style Guide](https://google.github.io/styleguide/pyguide.html) * vscode常用套件? * `even better toml`, `reload`, `remote ssh`, `Docker`, ... * jupyter notebook: * 搜尋notebook format on save, 把它打勾 * type hint: * 搜尋type hint, 開啟python analysis -> inlay hint (將function return types那一行打勾) --- # GPU環境設置 * CPU vs. GPU * CPU: 時脈高, 核心數少 * GPU: 時脈低, 核心數多 * NVIDIA GPU生態系 ![截圖 2023-11-04 上午8.40.26.png](https://hackmd.io/_uploads/rkRZmM77a.png) * 可能的環境設置方式 * 使用Docker 1. Linux → NVIDIA GPU Driver → Docker → 啟動[TensorFlow](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow), [PyTorch](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)等開發框架的 Docker image 2. Windows → NVIDIA GPU Driver → [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) → Docker → 啟動[TensorFlow](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow), [PyTorch](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)等開發框架的 Docker image * 直接於系統內安裝 * Windows → NVIDIA Driver (NVIDIA顯卡驅動) → CUDA (NVIDIA顯卡軟體開發框架) → cuDNN (用於NVIDIA顯卡運算的神經網路函式庫) → TensorFlow。 ## GPU相關套件 - 於 Windows 內直接安裝安裝順序: NVIDIA Driver (NVIDIA顯卡驅動) → CUDA (NVIDIA顯卡軟體開發框架) → cuDNN (用於NVIDIA顯卡運算的神經網路函式庫) → TensorFlow。 0. 安裝[NVIDIA Driver](https://www.nvidia.com/Download/index.aspx) (此步驟可略過，因為教室的機器應該已經安裝好NVIDIA驅動程式) 1. 啟動 `NVSMI` (NVIDIA System Management Interface) 開啟終端機 (如GIT BASH或Windows命令提示字元), 輸入`nvidia-smi`後, 按Enter, 即可顯示出GPU使用率等資訊。此步驟若能成功執行，也代表驅動程式應已正常安裝。 2. 安裝[CUDA v11.6.1](https://developer.nvidia.com/cuda-toolkit)。[[下載連結]](https://developer.download.nvidia.com/compute/cuda/11.6.1/local_installers/cuda_11.6.1_511.65_windows.exe) 3. 安裝[cuDNN v 8.4.0 (需與CUDA v11.6.1相容)](https://developer.nvidia.com/cudnn)。[[下載連結]](https://www.dropbox.com/s/vg4yp1cf5hvf2p3/cudnn-windows-x86_64-8.4.0.27_cuda11.6-archive.zip?dl=1) 將解壓縮後的cuDNN資料夾內的檔案, 逐一複製到```C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6```。 4. 執行以下指令，確定NVIDIA CUDA Compiler (NVCC)有被安裝: ```nvcc -V``` 如果有正常執行，表示CUDA應該可以使用。 5. 安裝TensorFlow等套件: 1. 執行`pip install -r requirements.txt` ``` matplotlib seaborn numpy>=1.23.4, <2.0.0 pandas>=1.5.3, <2.0.0 scipy>=1.9.1, <2.0.0 tensorflow<2.11 scikit-image>=0.20.0, <0.21.0 opencv-python>=4.6.0, <5.0.0 loguru>=0.6.0, <0.7.0 more-itertools>=8.14.0, <9.0.0 joblib>=1.2.0, <2.0.0 ``` 4. 檢查TensorFlow是否可把張量丟到GPU ```python import numpy as np import tensorflow as tf from loguru import logger if __name__ == "__main__": logger.info(tf.__version__) t = np.random.normal(0, 1, (3, 3)).astype(np.float32) # 以32 (bits), 也就是 32/8=4 (bytes) 來儲存一個浮點數。 # 上述建立了一個3x3亂數矩陣每個矩陣元素皆以4 bytes來儲存。 logger.info(t) logger.info(t.shape) logger.info(t.dtype) t = tf.constant(t) # 改以TensorFlow來儲存先前NumPy定義出來的張量 (順利的話會存在GPU上面的記憶體) logger.info(t.device) # 顯示張量存放位置 ``` 如有出現`GPU:0`字眼，表示GPU已可正常使用。 6. 修復TensorFlow Bug TensorFlow `v2.10.1`在Windows系統上使用卷基層做模型推論的時候會遇到錯誤 (與`zlibwapi.dll`相關)。如果你有遇到這個問題，可將: `C:\Program Files\NVIDIA Corporation\Nsight Systems 2021.5.2\host-windows-x64\zlib.dll` 更名，並且複製到 `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\zlibwapi.dll` 。複製完後，簡單將一個隨機張量丟進一個卷積層，即可確認問題是否解決。 ```python= import numpy as np from keras.layers import Conv2D from keras.models import Sequential if __name__ == '__main__': randData = np.random.normal(0, 1, (10, 5, 5, 3)) # normal分佈的亂數資料當input, 10個3D樣本 model = Sequential() model.add( Conv2D( filters=96, kernel_size=(3, 3), strides=(1, 1), padding="valid", input_shape=(5, 5, 3), ) ) print(model.predict(randData).shape) # 看輸出資料的形狀 ``` # 參考連結 * Design Pattern, Python basics: https://www.youtube.com/@ArjanCodes * Find state-of-art model implementations (if papers are with codes :arrow_right: :100:): https://paperswithcode.com * Understand CNN basics (:100: you must read this, understand every lines of words): https://cs231n.github.io * karparthy: https://www.youtube.com/results?search_query=karparthy * Learn Deep Learning basics (Wow! Learning by doing! :100:): https://zh.d2l.ai * Machine Learning basic theories (:100: for those who wants to understand every general bits of maths in Machine Learning): https://cs229.stanford.edu/notes2022fall/main_notes.pdf * Understand Backpropagation: https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf * Python Machine Learning (Sebastian Raschka) * Keras book: https://www.manning.com/books/deep-learning-with-python --- wengchihung@gmail.com

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.