Characteristics

• RNNs are a family of NNs for processing sequential data.
• An internal state that serves as a memory of what the network has seen in the past.
• An RNN shares the same weights accross several time steps .
• Regardless of the sequence length, the learned model has the same input size.
• It is possible to use the same transition function \(f\) with the same parameters at ever time \(t\).

Applications

• They allow to implement very flexible architecture designs.
• Can process sequences of variable length.
• Stock market price predictions
• Speech recognition
• Predicting trajectories

Simple recurrent neuron

• A recurrent neural network looks similar to a feedforward network except it also has connections pointing backwards.
• A simple recurrent neuron has the following components:
  1. An input.
  2. A single neuron that processes the output.
  3. One output connection.
  4. One connection that sends the output back to the neuron.

Recurrent neuron as memory cell

• A part of a neural network that preserves some state accross time steps is called a memory cell.
• The hidden state serves as a kind of lossy summary of the task-relevant aspects of the past sequence of inputs up to the current state.
• A single recurrent neuron, or a layer of recurrent neurons, is a basic cell or simply a cell.

Simple recurrent neuron

• It is possible "unroll" the recurrent neuron to represent the input and output at every time \(t\).
• It is always the same neuron!!! Only the representation changes.

Recurrent neuron as memory cell

\[ {\bf{s}}_{(t)} = f_W \left ( {\bf{s}}_{(t-1)}, {\bf{x}}_{(t)} \right) \]

\( {\bf{x}}_{(t)}\): Input.

\( {\bf{s}}_{(t-1)}, {\bf{s}}_{(t)}\): States at times \(t-1 \) and \( t \).

\( f_W \): Current state of the neuron depends on the previous state and the input through a function parameterized by \( W \) .

BPTT

• A forward pass through the unrolled network.
• The output sequence is evaluated using a cost function \(L \left ({\bf{Y}}_{(t_{min})},{\bf{Y}}_{(t_{min}+1)},\dots,{\bf{Y}}_{(t_{max})} \right) \), where \(t_{min}\) and \(t_{max}\) are the first and last ouput time steps.
• The gradients of the loss function are propagated backward through the unrolled network.
• The model parameters are updated using the gradients.
• Loss functions depend on the different RNN architectures.

Recurrent Layer

Recurrent Layer

Deep learning with RNNs

• A recurrent neural network can be made deep in many ways.
• Adding depthness from the input to the hidden state.
• From the previous hidden state to the next hidden state.
• From the hidden state to the output.
• Depthness can increase the capacity of the model needed in these different parts.

Deep learning with RNNs

• The hidden recurrent state can be broken down into groups organized hierarchically.
• Deeper computation (e.g., an MLP) can be introduced in the input-to-hidden, hidden-to-hidden, and hidden-to-ouput parts.
• Significant benefit can be obtained from decomposing the state of an RNN into multiple ways.
• Adding depth may hurt learning by making optimization difficult.

Ways of adding context

• A single vector \(x\) can be used as input context to condition sequence prediction.
• Can be added as an extra input at each time step or;
• As the initial state \(h^{(0)}\).
• Or be used as both of these previous possibilities.
• Context is used for tasks such as image captioning.

Characteristics

• Similar to the sequence model.
• It has an initial encoding stage with no output signal to predict.
• In the encoding stage, the input sequence is consumed to produce a hidden state.
• It the decoding stage, an output signal is produced, one timestep at a time, conditioned on the hidden state from the encoding stage .

Characteristics

• Two layers of hidden nodes are connected to input and output.
• The first has recurrent connections from the last time.
• The second has recurrent connections from the future.
• It requires a fixed point in the future and the past.

Characteristics

• It addresses the limitations of simple recurrent cells to represent long term dependencies in the data.
• It can be used as a simple cell.
• The key idea is that the network can learn what to store in the long-term state, what to throw away, and what to read from it.
• Training will converge faster for LSTMs.
• Several LSTM variants have been proposed.

Components

1. The forget gate \(f_{(t)} \) processes \( {\bf{h}}_{(t-1)} \) and \( {\bf{x}}_{(t)} \) to decide, from each variable, which information will be "forgotten" or "used" by the cell:
   
   \[ f_t = \sigma \left ( {\bf{W}}_{f} [{\bf{h}}_{(t-1)}, {\bf{x}}_{(t)}] + b_f \right ) \]

Components

2. The long term state candidate , \( {\bf{\tilde{C}}}_{(t)} \), is created by applying a tanh operator to the concatenation of the short term state and the input.:
   
   \[ \begin{align} {\bf{\tilde{C}}}_{(t)} &= tahn \left ( {\bf{W}}_{C} [{\bf{h}}_{(t-1)}, {\bf{x}}_{(t)}] + b_C \right ) \end{align} \]
3. The input gate \(i_{(t)} \) decides which values from \( {\bf{h}}_{(t-1)} \) and \( {\bf{x}}_{(t)} \) will be used to update \( {\bf{C}}_{(t)} \):
   
   \[ \begin{align} i_t &= \sigma \left ( {\bf{W}}_{i} [{\bf{h}}_{(t-1)}, {\bf{x}}_{(t)}] + b_i \right ) \\ \end{align} \]

Components

4. The long term state value \( {\bf{C}}_{(t)} \) is created as the sum of two terms. The product between the input gate and the long term state candidate, \( {\bf{\tilde{C}}}_{(t)} \); and the product between the forget gate and the previous short term state:
   
   \[ {\bf{C}}_{(t)} = i_t \cdot {\bf{\tilde{C}}}_{(t-1)} + f_t \cdot {\bf{C}}_{(t-1)} \]

Components

5. The output gate \(i_{(t)} \) controls which parts of the short-term state should be read and output (both to \( {\bf{y}}_{(t)} \) ) and \( {\bf{h}}_{(t)} \):
   
   \[ \begin{align} o_t &= \sigma \left ( {\bf{W}}_{o} [{\bf{h}}_{(t-1)}, {\bf{x}}_{(t)}] + b_o \right ) \\ \end{align} \]
6. The short-term state is the product between the output gate and a tanh operation on the long-term state:
   
   \[ \begin{align} {\bf{h}}_{(t)} &= o_t \cdot tahn({\bf{C}}_{(t)}) \end{align} \]

1. The forget gate \(f_{(t)} \) processes \( {\bf{h}}_{(t-1)} \) and \( {\bf{x}}_{(t)} \) to decide, from each variable, which information will be "forgotten" or "used" by the cell:
   
   \[ f_t = \sigma \left ( {\bf{W}}_{f} [{\bf{h}}_{(t-1)}, {\bf{x}}_{(t)}] + b_f \right ) \]
2. The long term state candidate , \( {\bf{\tilde{C}}}_{(t)} \), is created by applying a tanh operator to the concatenation of the short term state and the input:
   
   \[ \begin{align} {\bf{\tilde{C}}}_{(t)} &= tahn \left ( {\bf{W}}_{C} [{\bf{h}}_{(t-1)}, {\bf{x}}_{(t)}] + b_C \right ) \end{align} \]
3. The input gate \(i_{(t)} \) decides which values from \( {\bf{h}}_{(t-1)} \) and \( {\bf{x}}_{(t)} \) will be used to update \( {\bf{C}}_{(t)} \):
   
   \[ \begin{align} i_t &= \sigma \left ( {\bf{W}}_{i} [{\bf{h}}_{(t-1)}, {\bf{x}}_{(t)}] + b_i \right ) \\ \end{align} \]

Characteristics

• GRU is a simplified version of LSTM with a similar performance.
• Both state vectors are merged into a single output vector \( {\bf{h}}_{(t)} \) .
• A single update gate controller \(z\) controls both the forget state and the input state.
• If the update gate controller outputs a 1, the input gate is open and the forget gate is closed. If it outputs 0, the opposite happens.
• There is not output gate; the full state vector is output at every time step.