Recursive Neural Network


Recursive neural networks represent yet another generalization of recurrent networks, with a different kind of computational graph, which is structured as a deep tree, rather than the chain-like structure of RNNs. The typical computational graph for a recursive network is illustrated in figure below. Recursive neural networks were introduced by Pollack (1990) and their potential use for learning to reason was described by Bottou (2011). Recursive networks have been successfully applied to processing data structures as input to neural nets (Frasconi et al., 1997, 1998), in natural language processing (Socher et al., 2011a,c, 2013a) as well as in computer vision (Socher et al., 2011b).



One clear advantage of recursive nets over recurrent nets is that for a sequence of the same length $τ$, the depth (measured as the number of compositions of nonlinear operations) can be drastically reduced from $τ$ to $O(log τ )$, which might help deal with long-term dependencies. An open question is how to best structure the tree. One option is to have a tree structure which does not depend on the data, such as a balanced binary tree. In some application domains, external methods can suggest the appropriate tree structure. For example, when processing natural language sentences, the tree structure for the recursive network can be fixed to the structure of the parse tree of the sentence provided by a natural language parser (Socher et al., 2011a, 2013a). Ideally, one would like the learner itself to discover and infer the tree structure that is appropriate for any given input, as suggested by Bottou (2011).

A Recursive Neural Network is used for sentiment analysis in natural language sentences. It is one of the most important tasks of Natural language Processing (NLP), which identifies the writing tone and sentiments of the writer in a particular sentence. If a writer expresses any sentiment, basic labels about the writing tone are recognized. We want to identify the smaller components like nouns or verb phrases and order them in a syntactic hierarchy. For example, it identifies whether the sentence showcases a constructive form of writing or negative word choices.

A variable called 'score' is calculated at each traversal of nodes, telling us which pair of phrases and words we must combine to form the perfect syntactic tree for a given sentence.

Let us consider the representation of the phrase -- "a lot of fun" in the following sentence.

Programming is a lot of fun.

An RNN representation of this phrase would not be suitable because it considers only sequential relations. Each state varies with the preceding words' representation. So, a subsequence that doesn't occur at the beginning of the sentence can't be represented. With RNN, when processing the word 'fun,' the hidden state will represent the whole sentence.

However, with a Recursive Neural Network (RvNN), the hierarchical architecture can store the representation of the exact phrase. It lies in the hidden state of the node R_{a\ lot\ of\ fun}. Thus, Syntactic parsing is completely implemented with the help of Recursive Neural Networks.

Benefits of RvNNs for Natural Language Processing
The two significant advantages of Recursive Neural Networks for Natural Language Processing are their structure and reduction in network depth.

As already explained, the tree structure of Recursive Neural Networks can manage hierarchical data like in parsing problems.
 
Another benefit of RvNN is that the trees can have a logarithmic height. When there are O(n) input words, a Recursive Neural Network can represent a binary tree with height O(log\ n). This lessens the distance between the first and last input elements. Hence, the long-term dependency turns shorter and easier to grab.

Disadvantages of RvNNs for Natural Language Processing
The main disadvantage of recursive neural networks can be the tree structure. Using the tree structure indicates introducing a unique inductive bias to our model. The bias corresponds to the assumption that the data follow a tree hierarchy structure. But that is not the truth. Thus, the network may not be able to learn the existing patterns.

Another disadvantage of the Recursive Neural Network is that sentence parsing can be slow and ambiguous. Interestingly, there can be many parse trees for a single sentence.

Also, it is more time-consuming and labor-intensive to label the training data for recursive neural networks than to construct recurrent neural networks. Manually parsing a sentence into short components is more time-consuming and tedious than assigning a label to a sentence.

Comments

Post a Comment

Popular posts from this blog

NEURAL NETWORKS AND DEEP LEARNING CST 395 CS 5TH SEMESTER HONORS COURSE NOTES - Dr Binu V P, 9847390760

About Me Syllabus Previous Year Question Papers CST 395 Neural Network and Deep Learning Module 1 ( Basics of Machine Learning) Overview of Machine Learning Machine Learning Algorithm Linear Regression Capacity, Overfitting and Underfitting Regularization Hyperparameters and Validation Sets Estimators, Bias , Variance and Consistency Challenges In Machine Learning Linear and Logistic Regression ( Python code) Performance Measures Differentiation of  sigmoid and cross entropy function Gradient Descent, Stochastic, Batch and mini batch gradient descent Regression with Gradient Descent Module-2 (Neural Networks) Introduction to Neural Networks Application of Neural Networks Basic Architecture- Single Layer Neural Network Power of Function Composition- Non Linear Activation XOR Activation Functions Choice of activation and loss functions Multi Layer Neural Network, Back Propagation Back Propagation- Example Implementation of a two layer network(XOR) with sigmoid activation functio...
Read more

Syllabus CST 395 Neural Network and Deep Learning

Syllabus Module - 1 (Basics of Machine Learning ) Machine Learning basics - Learning algorithms - Supervised, Unsupervised, Reinforcement, overfitting, Underfitting, Hyper parameters and Validation sets, Estimators -Bias and Variance. Challenges in machine learning. Simple Linear Regression, Logistic Regression, Performance measures - Confusion matrix, Accuracy, Precision, Recall, Sensitivity, Specificity, Receiver Operating Characteristic curve( ROC), Area Under Curve(AUC). Module -2 (Neural Networks ) Introduction to neural networks -Single layer perceptrons, Multi Layer Perceptrons (MLPs), Representation Power of MLPs, Activation functions - Sigmoid, Tanh, ReLU, Softmax. Risk minimization, Loss function, Training MLPs with backpropagation, Practical issues in neural network training - The Problem of Overfitting, Vanishing and exploding gradient problems, Difficulties in convergence, Local and spurious Optima, Computational Challenges. Applications of neural networks. Module 3 (D...
Read more

Introduction to neural networks

Image
Artificial neural networks are popular machine learning techniques that simulate the mechanism of learning in biological organisms. The human nervous system contains cells, which are referred to as neurons . The neurons are connected to one another with the use of axons and dendrites , and the connecting regions between axons and dendrites are referred to as synapses .The strengths of synaptic connections often change in response to external stimuli. This change is how learning takes place in living organisms. This biological mechanism is simulated in artificial neural networks, which contain computation units that are referred to as neurons . The computational units are connected to one another through weights, which serve the same role as the strengths of synaptic connections in biological organisms. Each input to a  neuron is scaled with a weight, which affects the function computed at that unit. The following figure shows both the biological and artificial neural netw...
Read more