Bidirectional Recurrent Networks

One disadvantage of recurrent networks is that the state at a particular time unit only has knowledge about the past inputs up to a certain point in a sentence, but it has no knowledge about future states. In certain applications like language modeling, the results are vastly improved with knowledge about both past and future states. A specific example is handwriting recognition in which there is a clear advantage in using knowledge about both the past and future symbols, because it provides a better idea of the underlying context.

In the bidirectional recurrent network, we have separate hidden states $\bar{h}_t^{(f)}$  and  $\bar{h}_t^{(b)}$ for the forward and backward directions. The forward hidden states interact only with each other and the same is true for the backward hidden states. The main difference is that the forward states interact in the forwards direction, while the backwards states interact in the backwards direction. Both  $\bar{h}_t^{(f)}$  and  $\bar{h}_t^{(b)}$ however, receive input from the same vector $\bar{x}_t$(e.g., one-hot encoding of word) and they interact with the same output vector $\hat{y}_t$. An example of three time-layers of the bidirectional RNN is shown in Figure below.

There are several applications in which one tries to predict the properties of the current tokens, such as the recognition of the characters in a handwriting sample, a the parts of speech in a sentence, or the classification of each token of the natural language. In general, any property of the current word can be predicted more effectively using this approach, because it uses the context on both sides. For example, the ordering of words in several languages is somewhat different depending on grammatical structure. Therefore, a bidirectional recurrent network often models the hidden representations of any specific point in the sentence in a more robust way with the use of backwards and forwards states, irrespective of the specific nuances of language structure. In fact, it has increasingly become more common to use bidirectional recurrent networks in various language-centric applications like speech recognition.

For example, in speech recognition, the correct interpretation of the current sound as a phoneme may depend on the next few phonemes because of co-articulation and potentially may even depend on the next few words because of the linguistic dependencies between nearby words: if there are two interpretations of the current word that are both acoustically plausible, we may have to look far into the future (and the past) to disambiguate them. This is also true of handwriting recognition and many other sequence-to-sequence learning tasks, described in the next section.Bidirectional recurrent neural networks (or bidirectional RNNs) were invented to address that need (Schuster and Paliwal, 1997). They have been extremely successful (Graves, 2012) in applications where that need arises, such as handwriting recognition (Graves et al., 2008; Graves and Schmidhuber, 2009), speech recognition (Graves and Schmidhuber, 2005; Graves et al., 2013) and bioinformatics (Baldi et al., 1999).

In the case of the bidirectional network, we have separate forward and backward parameter matrices. The forward matrices for the input-hidden, hidden-hidden, and hidden-output interactions are denoted by $W_{xh}^{(f)}, W_{hh}^{(f)}, W_{hy}^{(f)}$,  respectively. The backward matrices for the input-hidden, hidden-hidden, and hidden-output interactions are denoted by $W{xh}^{(b)}, W{hh}^{(b)}, W{hy}^{(b)}$, respectively.

The recurrence conditions can be written as follows:

$\bar{h}_t^{(f)}=tanh(W_{xh}^{(f)}\bar{x}_t+W_{hh}^{(f)}\bar{h}_{t-1}^{(f)}) $

$\bar{h}_t^{(b)}=tanh(W_{xh}^{(b)}\bar{x}_t+W_{hh}^{(b)}\bar{h}_{t+1}^{(b)})$

$\bar{y}_t=W_{hy}^{(f)}\bar{h}_t^{(f)}+W_{hy}^{(b)}\bar{h}_t^{(b)}$

It is easy to see that the bidirectional equations are simple generalizations of the conditions used in a single direction. It is assumed that there are a total of T time-stamps in the neural network shown above, where $T$ is the length of the sequence. One question is about the forward input at the boundary conditions corresponding to $t = 1$ and the backward input at $t = T$, which are not defined. In such cases, one can use a default constant value of 0.5 in each case, although one can also make the determination of these values as a part of the learning process.

An immediate observation about the hidden states in the forward and backwards direction is that they do not interact with one another at all. Therefore, one could first run the sequence in the forward direction to compute the hidden states in the forward direction, and then run the sequence in the backwards direction to compute the hidden states in the backwards direction. At this point, the output states are computed from the hidden states in the two directions.

After the outputs have been computed, the backpropagation algorithm is applied to compute the partial derivatives with respect to various parameters. First, the partial derivatives are computed with respect to the output states because both forward and backwards states point to the output nodes. Then, the backpropagation pass is computed only for the forward hidden states starting from $t = T$down to $t = 1$. The backpropagation pass is finally computed for the backwards hidden states from $t = 1$ to $t = T$. Finally, the partial derivatives with respect to the shared parameters are aggregated. Therefore, the BPTT (backpropagation through time) algorithm can be modified easily to the case of bidirectional networks. 

We summarize the BPTT algorithm as follows:

(i) We run the input sequentially in the forward direction through time and compute the errors (and the negative-log loss of softmax layer) at each time-stamp.

(ii) We compute the gradients of the edge weights in the backwards direction on the unfurled network without any regard for the fact that weights in different time layers are shared. In other words, it is assumed that the weights $W_{xh}^{(t)},W_{hh}^{(t)}$ and  $W_{hy}^{(t)}$ time-stamp $t$ are distinct from other time-stamps. As a result, one can use conventional backpropagation to compute $\frac{∂L}{∂W_{xh}^{(t)}},\frac{∂L}{∂W_{hh}^{(t)}}$ and $\frac{∂L}{∂W_{hy}^{(t)}}$

Note that we have used matrix calculus notations where the derivative with respect to a matrix is defined by a corresponding matrix of element-wise derivatives.

(iii) We add all the (shared) weights corresponding to different instantiations of an edge in time. In other words, we have the following:

$\frac{\partial L}{\partial W_{xh}}=\sum_{t=1}^T \frac{\partial L}{\partial W_{xh}^{(t)}}$

$\frac{\partial L}{\partial W_{hh}}=\sum_{t=1}^T \frac{\partial L}{\partial W_{hh}^{(t)}}$

$\frac{\partial L}{\partial W_{hy}}=\sum_{t=1}^T \frac{\partial L}{\partial W_{hy}^{(t)}}$

The BPTT algorithm is modified as follows

1. Compute forward and backwards hidden states in independent and separate passes.

2. Compute output states from backwards and forward hidden states.

3. Compute partial derivatives of loss with respect to output states and each copy of the output parameters.

4. Compute partial derivatives of loss with respect to forward states and backwards states independently using backpropagation. Use these computations to evaluate partial derivatives with respect to each copy of the forwards and backwards parameters.

5. Aggregate partial derivatives over shared parameters.

In general, bidirectional RNNs work well in applications where the predictions are based on bidirectional context. Examples of such applications include handwriting recognition and speech recognition, in which the properties of individual elements in the sequence depend on those on either side of it. For example, if a handwriting is expressed in terms of the strokes, the strokes on either side of a particular position are helpful in recognizing the particular character being synthesized. Furthermore, certain characters are more likely to be adjacent than others.

A bidirectional neural network achieves almost the same quality of results as using an ensemble of two separate recurrent networks, one in which the input is presented in original form and the other in which the input is reversed. The main difference is that the parameters of the forwards and backwards states are trained jointly in this case. However, this integration is quite weak because the two types of states do not interact directly with one another.