Strides

 There are other ways in which convolution can reduce the spatial footprint of the image (or hidden layer). The above approach performs the convolution at every position in the spatial location of the feature map. However, it is not necessary to perform the convolution at every spatial position in the layer. One can reduce the level of granularity of the convolution by using the notion of strides. The description above corresponds to the case when a stride of 1 is used. When a stride of $S_q$ is used in the $q$th layer, the convolution is performed at the locations $1, S_q + 1, 2S_q + 1$, and so on along both spatial dimensions of the layer. The spatial size of the output on performing this convolution1 has height of $(L_q − F_q)/S_q + 1$ and a width of $(B_q − F_q)/S_q + 1$. As a result, the use of strides will result in a reduction of each spatial dimension of the layer by a factor of approximately $S_q$ and the area by $S^2_q$ ,although the actual factor may vary because of edge effects. It is most common to use a stride of 1, although a stride of 2 is occasionally used as well. It is rare to use strides more than 2 in normal circumstances. Even though a stride of 4 was used in the input layer of the winning architecture  of the ILSVRC competition of 2012, the winning entry in the subsequent year reduced the stride to 2  to improve accuracy. Larger strides can be helpful in memory-constrained settings or to reduce overfitting if the spatial resolution is unnecessarily high. 

Strides have the effect of rapidly increasing the receptive field of each feature in the hidden layer, while reducing the spatial footprint of the entire layer. An increased receptive field is useful in order to capture a complex feature in a larger spatial region of the image. As we will see later, the hierarchical feature engineering process of a convolutional neural network captures more complex shapes in later layers. Historically, the receptive fields have been increased with another operation, known as the max-pooling operation. In recent years, larger strides have been used in the  max-pooling operations.

Typical Settings

It is common to use stride sizes of 1 in most settings. Even when strides are used, small strides of size 2 are used. Furthermore, it is common to have $L_q = B_q$. In other words, it is desirable to work with square images. In cases where the input images are not square, preprocessing is used to enforce this property. For example, one can extract square patches of the image to create the training data. 

The number of filters in each layer is often a power of 2, because this often results in more efficient processing. Such an approach also leads to hidden layer depths that are powers of 2. Typical values of the spatial extent of the filter size (denoted by Fq) are 3 or 5. In general, small filter sizes often provide the best results, although some practical challenges exist in using filter sizes that are too small. Small filter sizes typically lead to deeper networks (for the same parameter footprint) and therefore tend to be more powerful. In fact, one of the top entries in an ILSVRC contest, referred to as VGG , was the first to experiment with a spatial filter dimension of only $F_q = 3$ for all layers, and the approach was found to work very well in comparison with larger filter sizes.

Use of Bias

As in all neural networks, it is also possible to add biases to the forward operations. Each unique filter in a layer is associated with its own bias. Therefore, the $p$th filter in the $q$th layer has bias $b(p,q)$. 

When any convolution is performed with the $p$th filter in the $q$th layer, the value of $b(p,q)$ is added to the dot product. The use of the bias simply increases the number of parameters in each filter by 1, and therefore it is not a significant overhead. Like all other parameters, the bias is learned during backpropagation. One can treat the bias as a weight of a connection whose input is always set to +1. This special input is used in all convolutions, irrespective of the spatial location of the convolution. Therefore, one can assume that a special pixel appears in the input whose value is always set to 1. Therefore, the number of input features in the $q$th layer is $1 + L_q × B_q × d_q$. This is a standard feature-engineering trick that is used for handling bias in all forms of machine learning.

Comments

Post a Comment

Popular posts from this blog

NEURAL NETWORKS AND DEEP LEARNING CST 395 CS 5TH SEMESTER HONORS COURSE NOTES - Dr Binu V P, 9847390760

About Me Syllabus Previous Year Question Papers CST 395 Neural Network and Deep Learning Module 1 ( Basics of Machine Learning) Overview of Machine Learning Machine Learning Algorithm Linear Regression Capacity, Overfitting and Underfitting Regularization Hyperparameters and Validation Sets Estimators, Bias , Variance and Consistency Challenges In Machine Learning Linear and Logistic Regression ( Python code) Performance Measures Differentiation of  sigmoid and cross entropy function Gradient Descent, Stochastic, Batch and mini batch gradient descent Regression with Gradient Descent Module-2 (Neural Networks) Introduction to Neural Networks Application of Neural Networks Basic Architecture- Single Layer Neural Network Power of Function Composition- Non Linear Activation XOR Activation Functions Choice of activation and loss functions Multi Layer Neural Network, Back Propagation Back Propagation- Example Implementation of a two layer network(XOR) with sigmoid activation functio...
Read more

Syllabus CST 395 Neural Network and Deep Learning

Syllabus Module - 1 (Basics of Machine Learning ) Machine Learning basics - Learning algorithms - Supervised, Unsupervised, Reinforcement, overfitting, Underfitting, Hyper parameters and Validation sets, Estimators -Bias and Variance. Challenges in machine learning. Simple Linear Regression, Logistic Regression, Performance measures - Confusion matrix, Accuracy, Precision, Recall, Sensitivity, Specificity, Receiver Operating Characteristic curve( ROC), Area Under Curve(AUC). Module -2 (Neural Networks ) Introduction to neural networks -Single layer perceptrons, Multi Layer Perceptrons (MLPs), Representation Power of MLPs, Activation functions - Sigmoid, Tanh, ReLU, Softmax. Risk minimization, Loss function, Training MLPs with backpropagation, Practical issues in neural network training - The Problem of Overfitting, Vanishing and exploding gradient problems, Difficulties in convergence, Local and spurious Optima, Computational Challenges. Applications of neural networks. Module 3 (D...
Read more

Introduction to neural networks

Image
Artificial neural networks are popular machine learning techniques that simulate the mechanism of learning in biological organisms. The human nervous system contains cells, which are referred to as neurons . The neurons are connected to one another with the use of axons and dendrites , and the connecting regions between axons and dendrites are referred to as synapses .The strengths of synaptic connections often change in response to external stimuli. This change is how learning takes place in living organisms. This biological mechanism is simulated in artificial neural networks, which contain computation units that are referred to as neurons . The computational units are connected to one another through weights, which serve the same role as the strengths of synaptic connections in biological organisms. Each input to a  neuron is scaled with a weight, which affects the function computed at that unit. The following figure shows both the biological and artificial neural netw...
Read more