ImageNet Classification with Deep Convolutional Neural Networks(翻譯)

# ImageNet Classification with Deep Convolutional Neural Networks(翻譯) ###### tags: `AlexNet` `CNN` `論文翻譯` `deeplearning` >[name=Shaoe.chen] [time=Thu, Feb 11, 2020] [TOC] ## 說明區塊如下分類，原文區塊為藍底，翻譯區塊為綠底，部份專業用語翻譯參考國家教育研究院 :::info 原文 ::: :::success 翻譯 ::: :::warning 任何的翻譯不通暢部份都請留言指導 ::: :::danger * [paper hyperlink](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf) ::: ## Abstract :::info We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. ::: :::success 我們訓練一個大型的深度卷積神經網路，在ImageNet LSVRC-2010競賽中將120萬張高解析度影像分類為1000個不同類別。在測試資料上，我們的top-1與top-5誤差率來到37.5%與17.0%，這比之前的[最新技術](http://terms.naer.edu.tw/detail/2913414/)還要好的多。這個神經網路擁有6,000萬個參數，與650,000個神經元，包含五個卷積層，其中一些後面接著max-pooling層，三個全連接層，最後是1000-way的softmax。為了讓訓練更快，我們使用非飽合神經元與非常高效的卷積操作的GPU實現。為了減少全連接層過擬合，我們採用一個最近開發的正規化方法，稱為"dropout"，這方法被證明非常有效。我們還在ILSVRC-2012競賽中加入這個模型的變體，這模型的top-5測試誤差率來到15.3%，而第二名的測試誤差率為26.2%。 ::: ## 1 Introduction :::info Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB \[16\], Caltech-101/256 \[8, 9\], and CIFAR-10/100 \[12\]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance \[4\]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. \[21\]), but it has only recently become possible to collect labeled datasets with millions of images. The new larger datasets include LabelMe \[23\], which consists of hundreds of thousands of fully-segmented images, and ImageNet \[6\], which consists of over 15 million labeled high-resolution images in over 22,000 categories. ::: :::success 目前的物體辨識方法主要使用機器學習方法。要增強其效能，我們可以搜集大量資料集，學習更強大的模型，並使用更好的技術來避免過擬合。直到最近，標記影像的資料集相對較小 - 大約上萬的影像(例如，NORB \[16\]與Caltech-101/256 \[8, 9\]與CIFAR-10/100 \[12\])。這種大小的資料集可以很好的解決簡單的辨識任務，特別是如果它們使用label-preserving轉換來增強的話。舉例來說，MNIST數字辨識任務上，當前最佳的誤差率(<0.3%)已經接近人類水平\[4\]。但實際環境中的物體表現出相當大的可變性，因此要學習辨識它們，就必須使用更大的資料集。確實，小型影像資料集的缺點已經得到廣泛的認知(Pinto et al. \[21\])，但是，是一直到最近才有可能收集到數百萬帶標記影像的資料集。新的更大型的資料集包含LabelMe \[23\]，由數十萬完全分段的影像所組成，以及ImageNet \[6\]，組1,500百萬張帶標記的高解析度影像所組成，超過22,000個類別。 ::: :::info To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models \[16, 11, 13, 18, 15, 22, 26\]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse. ::: :::success 要從數百萬張影像中瞭解關於數千個物體，我們需要一個擁有大型學習能力的模型。然而，物體辨識任務的巨大複雜性意味著，即使資料集如ImageNet這麼大型也無法解決這個問題，所以我們的模型應該也有很多的先驗知識來彌補我們所沒有的資料。卷積神經網路(CNNs)構成這類的模型\[16, 11, 13, 18, 15, 22, 26\]。它們的容量可以透過改變它們的深度與寬度來控制，而且它們也對影像的性質(即，統計的[穩定性](http://terms.naer.edu.tw/detail/958036/)以及像素[相依性](http://terms.naer.edu.tw/detail/6602770/)的局部性)做出強力，幾乎正確的假設。因此，對比於帶有類似大小層的標準[前饋](http://terms.naer.edu.tw/detail/2455027/)神經網路，CNNs擁有更少的連接以及更少的參數量，而且更容易訓練，而它們的理論最佳效能可能只有稍差一點。 ::: :::info Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting. ::: :::success 儘管CNNs擁有十分吸引人的特性，儘管其本地架構相對有效率，但他們在大規模應用於高解析度影像時仍然存在著令人望之卻步的昂貴成本。幸運的是，目前的GPUs與高度最佳化的2D卷積實現的結合，足以促進有趣的大型CNN的訓練，而且近來的資料集，像是ImageNet，包含足夠訓練這類模型的已標記樣本而不會嚴重的過擬合。 ::: :::info The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions \[2\] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly^1^ . Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance. ::: :::success 這篇論文最主要的貢獻如下：我們在ILSVRC-2010與ILSVRC-2012競賽\[2\]中使用ImageNet的子集，訓練迄今為止曘大的卷積神經網路之一，而且在這些資料集上取得至今為止報告的最佳結果。我們寫了一個高度最佳化的2D卷積實現以及其它訓練卷積神經網路中的固有操作，我們將公開提供^1^。我們的網路包含許多新的特殊功能，這些功能可以提高效能，而且可以減少訓練時間，相關細節請見第3節。即使擁有120萬個已標記訓練資料，我們的網路大小造成的過擬合是一個嚴重的問題，因此我們使用多個有效的技術來避免過擬合，相關細節請見第4節。我們最後的網路包含五層卷積與三層全連接層，這深度似乎很重要：我們發現，移除任何一層卷積層(每個卷積層都包含模型參數1%不到)都會導致效能降低。 ::: :::info In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available. ::: :::success 最後，網路大小主要受限於當前GPUs的可用記憶體以及我們能接受的訓練時間。我們的網路需要五到六天的時間訓練(在兩張GTX 580 3GB GPUs上)。我們所有的實驗都說明著，只要有更快的GPUs跟更大的資料集就可以改善結果。 ::: ## 2 The Dataset :::info ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images. ::: :::success ImageNet是一個超過1,500萬張已標記的高解析度影像資料集，大約屬於22,000個類別。這些影像是從網路上收集的，而且人工透過Amazon的工具Mechanical Turk crowd-sourcing來標記。從2010年開始，做為Pascal視覺物體挑戰賽的一部份，每年舉辦一次名稱ImageNet Large-Scale Visual Recognition Challenge (ILSVRC)的比賽。ILSVRC使用ImageNet的子集，1000個類別中每一個類別大約1000張影像。總共大約120萬張訓練影像，50,000張驗證影像，以及150,000張測試影像。 ::: :::info ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model. ::: :::success ILSVRC-2010是ILSVRC唯一一個可用測試集標記的版本，因此這是我們多數實驗使用的版本。因為我們在 ILSVRC-2012競賽中加入我們的模型，第6節我們會報告在這版本資料集上的結果，而該測試集標記是不可用的。在ImageNet上，[通常](http://terms.naer.edu.tw/detail/966494/)會報告兩個誤差率，top-1與top-5，其中top-5的誤差率是測試影像中正確的標記不在模型認為前五個最可能的標記中的部份。 ::: :::info ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of 256 × 256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256×256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels. ::: :::success ImageNet由各種解析度的影像所組成，而我們的系統需要一個固定輸入的維度。因此我們將影像降採樣到固定的解析度256 x 256。給定一個矩型的影像，首先我們對影像做縮放，以便短邊長度為256，然後從縮放後的影像中間[裁出](https://tw.voicetube.com/definition/crop%20out)一個256 x 256的區塊。除了每一個像素減去整個資料集的mean activity之外，我們並沒有以其方式對影像做預處理。因此，我們以原始像素的RGB值訓練網路。 ::: ## 3 The Architecture :::info The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first. ::: :::success 網路的架構總結在圖2。包含八層學習層 - 五層卷積與三層全連接層。下面我們將說明我們的網路架構中[新穎](http://terms.naer.edu.tw/detail/5451445/)或特殊的功能。第3.1-3.4節的排序是根據我們評估其重要性而排序，因此第一個是最重要的。 ::: :::info ![](https://i.imgur.com/BhxIeZ8.png) Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264–4096–4096–1000. 圖2：我們CNN架構的圖示明顯指出兩個GPUs之間的權責劃分。一個GPU執行圖形頂部的圖層部份，另一個GPU執行圖形底部的圖層部份。GPUs僅在部份層之間通信。網路的輸入為150,528維，網路其餘層的神經元數量為253,000-186,624-64,896-64,896-43,264-4096-4096-1000。 ::: ### 3.1 ReLU Nonlinearity :::info The standard way to model a neuron’s output $f$ as a function of its input $x$ is with $f(x) = tanh(x)$ or $f(x) = (1 + e^{-x})^-1$. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity $f(x) = max(0, x)$. Following Nair and Hinton \[20\], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models. ::: :::success 建構神經元的輸出$f$做為其輸入$x$函數的標準方法是使用$f(x) = tanh(x)$ or $f(x) = (1 + e^{-x})^-1$。就以梯度下降的訓練時間而言，這些飽合的[非線性](http://terms.naer.edu.tw/detail/58014/)比非飽合的[非線性](http://terms.naer.edu.tw/detail/58014/)$f(x) = max(0, x)$要慢的多。依據Nair and Hinton \[20\]，我們將這些[非線性](http://terms.naer.edu.tw/detail/58014/)的神經元稱為Rectified Linear Units (ReLUs)。深度卷積神經網路使用ReLUs訓練比使用tanh單元要快上幾倍。這在圖1中得到證明，該圖說明對於特定的四層卷積網路在CIFAR-10上的訓練誤差到達25%所需的迭代次數。這圖說明，如果我們使用傳統飽合神經元模型，那我們就無法以這麼大的神經網路進行實驗。 ::: :::info ![](https://i.imgur.com/fV2bgTb.png) Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rates for each network were chosen independently to make training as fast as possible. No regularization of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consistently learn several times faster than equivalents with saturating neurons. 圖1：帶ReLUs(實線)的四層卷積網路在CIFAR-10上的訓練誤差率到達25%比帶tanh神經元的相同網路(虛線)要快上六倍。每個網路的learn rate是獨立選擇，讓網路的訓練愈快愈好。沒有使用任何形式的正規化。這邊說明影響大小會隨網路架構不同而有所差異，但使用ReLUs的網路始終比帶飽合神經元的相同架構網路要快上數倍。 ::: :::info We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al. \[11\] claim that the nonlinearity $f(x) = \vert tanh(x) \vert$ works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets. ::: :::success 我們並不是第一個在CNNs中取代傳統神經元模型的人。像是，Jarrett et al. \[11\]就聲明，非線性函數$f(x) = \vert tanh(x) \vert$與其[對比度](http://terms.naer.edu.tw/detail/507747/)正規化，隨後是average pooling，這在Caltech-101資料集上執行的非常好。然而，在這資料集上，主要關心的是避免過擬合，因此他們所觀察的影響不同於我們使用ReLUs時所報告的擬合資料集的加速能力。快速學習對於在大型資料集上訓練的大型模型的效能上有很大的影響。 ::: ### 3.2 Training on Multiple GPUs :::info A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation. ::: :::success 只單GTX 580 GPU僅擁有3GB的記憶體，這限制了可以在顯卡上訓練最大網路的大小。事實證明，120萬個訓練樣本是足夠訓練網路，但太大而無法在一個GPU上訓練。因此，我們將網路[分佈](http://terms.naer.edu.tw/detail/6679177/)在兩張GPUs上。目前的GPUs特別合跨GPU[平行化](http://terms.naer.edu.tw/detail/6661483/)，因為它們直接讀寫彼此的記憶體，而不需要透過主機的記憶體。我們應用的[平行化](http://terms.naer.edu.tw/detail/6661483/)方案本質上是每個GPU都放一半的kernels(或神經元)，一個額外的技巧是：GPUs僅在部份層進行通信。這意味著，像是，第三層的kernel是從第二層的所有kernel maps取得輸入。但是，第四層的kernel僅從第三層中位於同一GPU上的那些kernel maps得到輸入。選擇[連接](http://terms.naer.edu.tw/detail/2113445/)模式是一個交叉驗證的問題，但這允許我們[精確](https://tw.voicetube.com/definition/precisely)調校通信數量，直到它的計算量是可接受的部份。 ::: :::info The resultant architecture is somewhat similar to that of the "columnar" CNN employed by Cire¸san et al. \[5\], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net^2^. ::: :::success 最終架構與Cire¸san et al. \[5\]所採用的"columnar"CNN有些相似，除了我們的columns不是獨立的(見圖2)。與一個GPU上訓練的每一個卷積層中一半的kernels網路相比，這個方案減少我們top-1與top-5的誤差率分別為1.7%與1.2%。雙GPU網路的訓練時間比單GPU網路的訓練時間要少一些^2^。 ::: :::info ^2^The one-GPU net actually has the same number of kernels as the two-GPU net in the final convolutional layer. This is because most of the net’s parameters are in the first fully-connected layer, which takes the last convolutional layer as input. So to make the two nets have approximately the same number of parameters, we did not halve the size of the final convolutional layer (nor the fully-conneced layers which follow). Therefore this comparison is biased in favor of the one-GPU net, since it is bigger than "half the size" of the two-GPU net. ::: :::success ^2^單GPU網路與雙GPU網路在最後卷積層確實有相同的kernels數量。這是因為多數的網路參數是在第一個全連接層中，這連接層以最後一層卷積層的輸出為輸入。因此，為了讓兩個網路有相近的參數量，我們將最後一層卷積的大小減半(接在後面的全連接層也沒有)。因此，這個比較是傾向於單GPU網路，因為它比雙GPU網的一半大小還要大。 ::: ### 3.3 Local Response Normalization :::info ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by $a^i_{x,y}$ the activity of a neuron computed by applying kernel $i$ at position $(x, y)$ and then applying the ReLU nonlinearity, the response-normalized activity $b^i_{x, y}$ is given by the expression $$ b^i_{(x, y)} = a^i_{(x, y)}/ \left( k + \alpha \sum_{j=\max(0, i-n/2)}^{\min(N - 1, i+n/2)} (a^j_{x, y})^2 \right)^\beta $$ where the sum runs over $n$ "adjacent" kernel maps at the same spatial position, and $N$ is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants $k$, $n$, $α$, and $\beta$ are hyper-parameters whose values are determined using a validation set; we used $k = 2, n = 5, α = 10^{−4}$ , and $\beta = 0.75$. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5). ::: :::success ReLUs擁有理想的屬性，也就是它們並不需要輸入正規化就可以預防飽合。如果至少有一些訓練樣本產生"正"輸入，那kernel就會進行學習。然而，我們仍然發現，下面的局部正規化方案有助於泛化。符號$a^i_{x,y}$表示kernel通過在位置$(x, y)$應用kernel$i$的[活動](http://terms.naer.edu.tw/detail/2110842/)(激活?啟動?)，然後應用ReLU非線性計算，響應正規化$b^i_{x, y}$的[活動](http://terms.naer.edu.tw/detail/2110842/)(激活?啟動?)由下面表達式給出： $$ b^i_{(x, y)} = a^i_{(x, y)}/ \left( k + \alpha \sum_{j=\max(0, i-n/2)}^{\min(N - 1, i+n/2)} (a^j_{x, y})^2 \right)^\beta $$ 其中求和運算在$n$個[相鄰的](http://terms.naer.edu.tw/detail/6617686/)kernel maps的同一空間位置上執行，$N$指該層中的所有kernels的總數。kernel maps的順序是當然是任意的，在訓練之前就已經決定好的。這種響應正規化實現了一種[橫向抑制](http://terms.naer.edu.tw/detail/6639262/)的形式，靈感來自真實神經元發現的類型，使用不同的核計算的神經元輸出之間產生的大型活動的競爭。常數$k$，$n$，$α$與$\beta$是超參數，其數值使用驗證集確定；我們使用$k = 2, n = 5, α = 10^{−4}$，以及$\beta = 0.75$。我們在某些層中應用ReLU非線性之後使用這個正規化(見章節3.5)。 ::: :::info This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. \[11\], but ours would be more correctly termed "brightness normalization", since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization^3^. ::: :::success 這個方案與Jarrett et al. \[11\]所提的局部對比正規化方案有些許相似，但我們的更正確的稱為"brightness normalization(亮度正規化)"，因為我們並沒有減去[平均活度量](http://terms.naer.edu.tw/detail/3488998/)。響應正規化使我們的top-1與top-5的誤差率各別減少1.4%與1.2%。我們還在CIFAR-10資料集上驗證這個方案的有效性：四層CNN沒有正規化來到13%的測試誤差率，而有正規化來到11%的測試誤差率^3^。 ::: :::info ^3^We cannot describe this network in detail due to space constraints, but it is specified precisely by the code and parameter files provided here: `http://code.google.com/p/cuda-convnet/`. ::: :::success ^3^因為空間的限制，我們並不能詳細的說明這個網路的細節，但這邊([連結](`http://code.google.com/p/cuda-convnet/`))提供的程式碼以及參數檔精確的[具體說明](https://tw.dictionary.search.yahoo.com/search?p=specified)該網路。 ::: ### 3.4 Overlapping Pooling :::info Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., \[17, 11, 4\]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size $z × z$ centered at the location of the pooling unit. If we set $s = z$, we obtain traditional local pooling as commonly employed in CNNs. If we set $s < z$, we obtain overlapping pooling. This is what we use throughout our network, with $s = 2$ and $z = 3$. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme $s = 2$, $z = 2$, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit. ::: :::success CNNs中的池化層(pooling layer)匯總相同kernel map中神經元[鄰域](http://terms.naer.edu.tw/detail/2120295/)群組的輸出。傳統上，由[相鄰](http://terms.naer.edu.tw/detail/6617686/)池單元匯總的[鄰域](http://terms.naer.edu.tw/detail/2120295/)不會重疊(如\[17, 11, 4\])。更精確的說，池化層可以被認為是由池化單元網格所組成，網格間隔為s像素，每個網格匯總池化單元中心位置大小$zxz$的[鄰域](http://terms.naer.edu.tw/detail/2120295/)。如果我們設置$s = z$，那我們就會獲得CNNs中常使用的傳統局部池化。如果設置我們$s < z$，我們就會得到重疊的池化。這也就是為什麼我們在整個網路中使用$s = 2$與$z = 3$的原因。與沒有重疊的方案比較，這個方案各別降低top-1與top-5的誤差率0.4%與0.3%($s = 2$，$z = 2$)，這產生相同的輸出維度。我們通常在訓練過程中觀察到，有重疊池化的模型比較不會過擬合。 ::: ### 3.5 Overall Architecture :::info Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully-connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution. ::: :::success 現在我們準備好說明我們CNN的總體架構。如圖2所示，網路包含八層帶權重層；前五層是卷積層，其餘三層是全連接層。最後一層全連接層的輸出被饋送到1000-way的softmax，這softmax在1000個類分標記上產生分佈。我們的網路最大化[多項式](http://terms.naer.edu.tw/detail/3216877/)邏輯迴歸目標，這等同於在預測分佈下最大化正確標記的對數機率的訓練樣本的平均值(?)。 ::: :::info The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully-connected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer. ::: :::success 第二、四、五層卷積的kernel僅連接到那些位於相同GPU上的前一層中的kernel maps(見圖2)。第三層的kernel連接第二層中所有的kernel maps。全連接層中的kernel則是連接到上一層中的所有kernel。響應正規化層接在第一與第二層卷積層後面。第3.4節所說的Max-pooling層則接在響應正規化層以及第五層卷積層後面。ReLU非線性則應用於每一個卷積層與全連接層的輸出。 ::: :::info The first convolutional layer filters the 224×224×3 input image with 96 kernels of size 11×11×3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 × 5 × 48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 × 3 × 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 × 3 × 192 , and the fifth convolutional layer has 256 kernels of size 3 × 3 × 192. The fully-connected layers have 4096 neurons each. ::: :::success 第一層卷積層過濾維度為224x224x3的輸入影像，96個大小11x11x3的kernel，stride為4(這是kernel map中[鄰域](http://terms.naer.edu.tw/detail/2120295/)神經元的感受域(receptive field)中心中間的距離)。第二層卷積層以第一層卷積層的輸出為輸入，並以256個大小5x5x48的kernel做過濾。第三、四、五卷積層彼此連接，而沒有中介的池化或正規化層。第三層卷積層擁有384個大小3x3x256的kernels連接到第二層卷積層的輸出(經過正規層與池化層)。第四層卷積層有384個大小3x3x192的kernels，第五層卷積層有256個大小3x3x192的kernels。每一層全連接層都擁有4,096個神經元。 ::: ## 4 Reducing Overfitting :::info Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting. ::: :::success 我們的神經網路架構擁有6,000萬個參數。雖然ILSVRC的1,000個類別讓每個訓練樣本從影像到標記的映射施加10bits的約束，但事實證明，這不足以學習如此多的參數而又沒有明顯的過擬合。下面，我們說明對抗過擬合的兩個方法。 ::: ### 4.1 Data Augmentation :::info The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., \[25, 4, 5\]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free. ::: :::success 降低影像資料過擬合最簡單與最常見的方法就是使用標記保留的[轉換](http://terms.naer.edu.tw/detail/958675/)來人工[放大](http://terms.naer.edu.tw/detail/2748704/)資料集(如\[25, 4, 5\])。我們採用兩種不同形式的資料增強，兩種形式都可以以很少的計算從原始影像生成轉換換的影像，因此，轉換後的影像並不需要保存在磁碟上。在我們的實作中，在GPU訓練上一批影像的時候，轉換的影像是以Python在CPU上生成的。因此，這些資料資料增強方案事實上，付出的計算成本是零的。 ::: :::info The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random 224 × 224 patches (and their horizontal reflections) from the 256×256 images and training our network on these extracted patches^4^ . This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches. ::: :::success 第一種形式的資料增強包含生成影像[平移](http://terms.naer.edu.tw/detail/955614/)與水平反射。我們透過從256x256影像中隨機提取224x224的區塊(及其水平反射)，然後以這些提取的區塊訓練網路^4^。這讓我們的訓練資料集增加了2048倍，儘管產生的訓練樣本想當然是高度的相互依賴。如果沒有這個方案，我們的網路會遭受嚴重的過擬合，這將強迫我們使用更小的網路。在測試的時候，網路透過提取五個224x224區塊(四個邊角區塊與一個中心區塊)以及它們的水平反射(因此總共十個區塊)做預測，然後將網路的softmax層對十個區塊的預測做平均。 ::: :::success ^4^這就是為什麼圖2中輸入影像的維度為224 x 224 x 3的原因。 ::: :::info The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel $I_{xy} = \left[I^R_{xy}, I^G_{xy}, I^B_{xy} \right]$ we add the following quantity: $$ \left[p_1, p_2, p_3 \right]\left[ \alpha_1 \lambda_1, \alpha_2 \lambda_2, \alpha_3 \lambda_3 \right]^T $$ where $p_i$ and $\lambda_i$ are $i$th eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and $\alpha_i$ is the aforementioned random variable. Each $\alpha_i$ is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over 1%. ::: :::success 第二種形式的資料增強包含變更訓練影像中RGB通道的[強度](http://terms.naer.edu.tw/detail/412190/)。具體而言，我們在整個ImageNet訓練集中對RBG像素值集執行PCA。這每一個訓練影像中，我們加入所找到的[主成份](http://terms.naer.edu.tw/detail/153880/)的倍數，其[大小](http://terms.naer.edu.tw/detail/2119304/)與相對應的[特徵值](http://terms.naer.edu.tw/detail/2115271/)乘上均值為0且[標準差](http://terms.naer.edu.tw/detail/1113798/)為0.1的高斯隨機變數與成[比例](http://terms.naer.edu.tw/detail/6437673/)。因此每一個RGB影像像素$I_{xy} = \left[I^R_{xy}, I^G_{xy}, I^B_{xy} \right]$，我們加入下面數量： $$ \left[p_1, p_2, p_3 \right]\left[ \alpha_1 \lambda_1, \alpha_2 \lambda_2, \alpha_3 \lambda_3 \right]^T $$ 其中$p_i$與$\lambda_i$是分別是RGB像素值的3x3[共變異方陣](http://terms.naer.edu.tw/detail/218090/)的第$i$個[特徵向量](http://terms.naer.edu.tw/detail/654204/)與[特徵值](http://terms.naer.edu.tw/detail/654205/)，而$\alpha_i$則是[前面提到](https://dictionary.cambridge.org/zht/%E8%A9%9E%E5%85%B8/%E8%8B%B1%E8%AA%9E-%E6%BC%A2%E8%AA%9E-%E7%B9%81%E9%AB%94/aforementioned?q=aforementioned+)的隨機變量。特定訓練影像的所有像素中的每一個$\alpha_i$僅取得一次，直到再次使用該影像進訓練，這時候才會重新取得一次。這個方案近似[擷取](http://terms.naer.edu.tw/detail/6597343/)自然影像的重要屬性，也就是，即使照明的強度與顏色的變化，對物體身份還是不變的。這方便讓top-1的誤差率降低1%以上。 ::: ### 4.2 Dropou :::info Combining the predictions of many different models is a very successful way to reduce test errors \[1, 3\], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called "dropout" \[10\], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks. ::: :::success 結合不同模型的預測是一個降低測試誤差非常成功的方法\[1, 3\]，但那對已經花費很多天訓練的大型神網路而言是非常昂貴的。但是，有一種非常有效率的結合模型的版本，而且訓練期間僅需要兩倍的成本。最近介紹的技術，稱為"dropout"\[10\]，包含將每個隱藏神經元以0.5的機率設置為零。以這種方法"dropped out"的神經元對正向傳遞沒有貢獻，也不會參與反向傳播。所以，當每一次出現輸入的時候，神經網路就會對不同架構做採樣，但所有的這些架構是共享權重的。這個技術降低了神經元的複雜[共同適應](http://terms.naer.edu.tw/detail/514540/)因為神經元並不會依賴於特定其它神經元的存在。因此，會迫使神經元學習更具魯棒性的特徵，這些特徵與其它神經元的許多不同隨機子集結合是有幫助的。在測試的時候，我們使用所有的神經元，但將它們的輸出乘上0.5，對取得由指數級的dropout網路所生成的預測分佈的幾何均值，這是合理的近似值。 ::: :::info We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge. ::: :::success 我們在圖2的前兩個全連接層使用dropout。如果沒有dropout，那我們的網路會呈現嚴重的過擬合。dropout大約需要兩倍的迭代數來收斂。 ::: ## 5 Details of learning :::info We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularizer: it reduces the model’s training error. The update rule for weight $w$ was $$ v_{i+1} := 0.9 \cdot v_i - 0.0005 \cdot \epsilon \cdot w_i - \epsilon \cdot \left\langle \dfrac{\partial L}{\partial w} \vert_{w_i} \right\rangle_{D_i} $$ $$ w_{i+1} := w_i + v_{i+1} $$ where $i$ is the iteration index, $v$ is the momentum variable, $\epsilon$ is the learning rate, and $\left\langle \dfrac{\partial L}{\partial w} \vert_{w_i} \right\rangle_{D_i}$ is the average over the $i$th batch $D_i$ of the derivative of the objective with respect to $w$, evaluated at $w_i$. ::: :::success 我們使用梯機梯度下降訓練模型，batch size為128，momentum為0.9，並設置weight decay為0.0005。我們發現，少量的weight decay對模型的學習是重要的。換句話說，這裡的weight decay不僅僅是一個正規化：它降低模型的訓練誤差。權重$w$的更新規則為 $$ v_{i+1} := 0.9 \cdot v_i - 0.0005 \cdot \epsilon \cdot w_i - \epsilon \cdot \left\langle \dfrac{\partial L}{\partial w} \vert_{w_i} \right\rangle_{D_i} $$ $$ w_{i+1} := w_i + v_{i+1} $$ 其中$i$是目前的迭代次數，$v$是變數momentum，$\epsilon$為learn rate，而$\left\langle \dfrac{\partial L}{\partial w} \vert_{w_i} \right\rangle_{D_i}$在$w_i$處估測相對於$w$的目標導數第$i$批$D_i$的平均值(?)。 ::: :::info We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0. ::: :::success 我們從均值為0且[標準差](http://terms.naer.edu.tw/detail/1113798/)為0.01的高斯分佈中初始化每一層的權重。我們使用常數1初始化第二、第四、第五層卷積層以及全連接隱藏層的神經偏差。透過提供ReLUs"正"輸入，這種初始化加速學習的早期階段。其餘層的神經偏差初始化為常數0。 ::: :::info We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs. ::: :::info 我們對所有層使用相同的learn rate，在整個訓練過程中手動調整。我們遵循[啟發法](http://terms.naer.edu.tw/detail/2117194/)，當驗證誤差率不再隨著當下的learn rate提高時，就將learn rate除以10。learn rate初始值為0.01，在終止之前降低三倍。我們透過120萬張影像對網路訓練大約90個循環(90次迭代?)，在兩塊NVIDIA GTX580 3GB GPUs上花了大概五到六天。 ::: ## 6 Results :::info Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%^5^ . The best performance achieved during the ILSVRC2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features \[2\], and since then the best published results are 45.7% and 25.7% with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features \[24\]. ::: :::success 我們在ILSVRC-2010的結果總結在Table 1。我們的網路top-1與top-5的測試集誤差率分別來到37.5%與17.0%。在ILSVRC2010競賽中，最佳效能分別為47.1%與28.2%，這方法是從訓練於不同特徵上的六個稀疏編碼模型產生的預測計算平均\[2\]，從那時候開始，發佈的最佳結果是45.7%與25.7%，這方法是從兩個訓練於Fisher Vectors(FVs)的分類器的預測值計算平均，這兩個分類器是由兩種densely-sampled特徵計算而來\[24\]。 ::: :::success ^5^;The error rates without averaging predictions over ten patches as described in Section 4.1 are 39.0% and 18.3%. ::: :::info |Model|Top-1|Top_5| | -------- | -------- | -------- | |*Sparse coding*|47.1%|28.2%| |*SIFT + FVs*|45.7%|25.7%| |CNN|37.5%|17.0%| Table 1: Comparison of results on ILSVRC2010 test set. In italics are best results achieved by others. Table 1：比較ILSVRC2010測試集的結果。斜體字是其它架構獲得的最佳結果。 ::: :::info We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than 0.1% (see Table 2). The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the predictions of five similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of 16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of 15.3%. The second-best contest entry achieved an error rate of 26.2% with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features \[7\]. ::: :::success 我們還在ILSVRC-2012競賽中帶入我們的模型，並在Table 2中報告結果。因為ILSVRC-2012測試集標記並不是公開可用的，我們無法報告嚐試過的所有模型的測試誤差率。這段落的其餘部份，我們交互使用驗證與測試誤差率，因為在我們的經驗中，它們的差異並沒有超過0.1%(見Table 2)。此論文介紹的CNN的top-5誤差率來到18.2%。平均五個相似CNNs的預測得到的誤差率為16.4%。訓練一個CNN，在最後一個池化層上加入一個額外的第六個卷積層，對整個ImageNet~(2011秋天發佈)~分類(1500萬張影像，22000個類別)，然後在ILSVRC-2012上"微調"，得到的誤差率為16.6%。平均兩個CNNs的預測()得到的誤差率為15.3%。整個2011秋季發佈期間以[上述的](https://tw.voicetube.com/definition/aforementioned)五個CNNs預訓練出來的兩個CNNs的預測做平均，得到誤差率為15.3%。第二好的競賽得到誤差率26.2%，以多個訓練於FVs的分類器(計算來自不同類型的densely-sampled特徵)的預測計算平均\[7\]。 ::: :::info |Model|Top-1(val)|Top-5(val)|Top-5(test)| | -------- | -------- | -------- | -------- | |*SIFT + FVs*|-|-|26.2%| |1 CNN|40.7&|18.2%|-| |5 CNNs|38.1%|16.4%|**16.4%**| |1 CNN\*|39.0%|16.6%|-| |7 CNNs\*|36.7%|15.4%|**15.3%**| Table 2: Comparison of error rates on ILSVRC-2012 validation and test sets. In italics are best results achieved by others. Models with an asterisk\* were "pre-trained" to classify the entire ImageNet 2011 Fall release. See Section 6 for details. Table 2：ILSVRC-2012競賽中驗證集與測試集的誤差率。斜體字是其它架構的最佳結果。帶星號\*的模型是"預訓練"以分類整個ImageNet(2011秋季發佈)。細節請見第6節。 ::: :::info Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are 67.4% and 40.9%, attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are 78.1% and 60.9% \[19\]. ::: :::success 最後，我們還報告了ImageNet 2009年秋季版本(10184個類別與890萬張影像)。在這資料集上，我們依循文獻中一半影像用於訓練，一半影像用於測試的慣例。因為並沒有建立測試集，我們分割的部份一定與之前作者不一樣，但這不會明顯影響結果。在這資料集上，我們top-1與top-5誤差率為67.4%與40.9%，實現上述的網路，但加入額外的部份，最後一個池化層上加入第六層卷積層。在這資料集上公佈的最佳結果是78.1%與60.9%\[19\]。 ::: ### 6.1 Qualitative Evaluations :::info ![](https://i.imgur.com/BJq2RLE.png) Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs). ::: :::success 圖3說明了網路的兩個資料連接層所學習的卷積kernel。網路已經學習到各種的頻率與方向選擇性的kernels，還有各種彩色的斑點。注意到，這是兩個GPUs所展現出來的特殊性，這是第3.5節所說的受限連接的結果。在GPU1上的kernel很大程度與顏色無關，而GPU2上的kernels很大程度是針對顏色的。這種特殊性會在每次執行期間發生，而且是獨立於任何特定的隨機權重初始化(就是無關的意思)。 ::: :::info Figure 3: 96 convolutional kernels of size 11×11×3 learned by the first convolutional layer on the 224×224×3 input images. The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details. 圖3：第一層卷積層在224x224x3的輸入影像上學習而的得96個大小11x11x3的kernels。前48個kernel是在GPU1上學習，後面48個在GPU2上學習。相關細節請參閱第6.1節。 ::: :::info In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph. ::: :::success 圖4左側面版中，我們透過計算八張測試影像的TOP-5預測來定性評估網路所學習到的東西。即使是[偏心](http://terms.naer.edu.tw/detail/3325953/)的物件(如左上角的[蟎](http://terms.naer.edu.tw/detail/422047/))，也可以被網路辨識到。多數的top-5標記看來都是合理的。舉例來說，只有其它類型的貓才會被考慮為豹的合理標記。在某些情況下([格柵](http://terms.naer.edu.tw/detail/3660496/)，櫻桃)照片的預期焦點確實模棱兩可。 ::: :::info ![](https://i.imgur.com/NpHt19Q.jpg) Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image. 圖4：(左)八個ILSVRC-2010測試影像與模型考慮的最有可能的五個標記。正確的標記寫在每一張影像下面，並且分配給正確的標記以紅色條顯示(如果它剛好在top-5)。(右)第一個column中五張ILSVRC-2010測試影像。其餘column顯示六張訓練影像，這些影像在最後一個隱藏層中生成特徵向量，其特徵向量與測試影像的特徵向量之間的歐幾里德距離最小。 ::: :::info Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material. ::: :::success 另一種探索網路視覺知識的方法就是考慮由最後4096維隱藏層像所引起的特徵[啟動](http://terms.naer.edu.tw/detail/628865/)(激活)。如果兩張影像產生的特徵啟動向量有較小的歐幾里德距離，那我們就可以說，神經網路較高層的部份將它們視為相似的。圖4說明五張測試影像集與六張訓練集影像根據這個測量方式取得彼此之間最相似的影像。注意到，在像素級別中，檢索訓練影像與第一個column內的查詢影像在L2上通常是不接近的。舉例來說，檢索的狗與大像以各種姿勢出現。我們在[補充材料](http://terms.naer.edu.tw/detail/1312509/)中提供更多測試影像結果。 ::: :::info Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying autoencoders to the raw pixels \[14\], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar. ::: :::success 透過使用兩個4096維向量([實質](http://terms.naer.edu.tw/detail/2123206/)向量)計算其歐幾里德距離是沒效率的，但可以利用訓練auto-encoder來壓縮這些向量為短二進制編碼來提高效率。比起應用自動編碼器在原始像素上相比，這應該會產生一個更好的影像檢索方法\[14\]，這方法不使用影像標記，因為有著檢索具有相似邊緣模式影像的趨勢，無論它們在語義上是否相似。 ::: ## 7 Discussion :::info Our results show that a large, deep convolutional neural network is capable of achieving record-breaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results. ::: :::success 我們的結果說明了，一個大型的深度卷積神經網路可以使用單純使用監督式學習在一個高挑戰性的資料集上得到破記錄的結果。值得注意的是，如果移除單個卷積層，那我們網路的效能會下降。舉例來說，移除任一個中間層的結果會讓網路top-1效能損失約2%。所以深度對於達到我們的結果真的非常重要。 ::: :::info To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images. ::: :::success 為了減化我們的實驗，我們並沒有使用任何的非監督式預訓練，即使我們希望這會有所幫助，特別是，如果我們獲得足夠的計算能力來明顯的加大網路的大小而又沒有獲得相對應增加的標記資料時。目前為止，因為我們加大了模型並訓練更長的時間，因此我們的結果有所改善，但為了匹配人類視覺系統的infero-temporal pathway，我們仍然有許多[量級](http://terms.naer.edu.tw/detail/261986/)工作要處理。最後，我們希望在視頻序列上使用非常大型的深度卷積網路，其時間架構提供非常有幫助的信息，這些信息在靜態影像中遺失或不是那麼明顯。 :::