# Recurrent Neural Networks

> [!NOTE]
>
> **In one minute**
>
> A recurrent neural network feeds the output of a unit back into itself, so its state at each time step depends on the current input and its previous state. Unrolled over time it is a chain of copies of the same unit, and it is closely related to a first-order differential equation.
>
> **Read first:** [Neural Networks](<https://dl.leima.is/deep-learning-fundamentals/neural-net/index.md>)

In the section [Neural Networks](<https://dl.leima.is/deep-learning-fundamentals/neural-net/index.md>), we discussed the feedforward neural network.

> [!NOTE]
>
> **Biological Neural Networks**
>
> Biological neural networks contain recurrent units. There are theories that employ recurrent networks to explain our memory[^/deep-learning-fundamentals/recurrent-neural-networks/#bib-Grossberg:2013].

## Recurrent Neural Network Architecture

A recurrent neural network (RNN) can be achieved by including loops in the network, i.e., the output of a unit is fed back to itself. As an example, we show a single unit in the following figure.

![Basic RNN](<https://dl.leima.is/deep-learning-fundamentals/assets/recurrent-neural-networks/rnn-simple.jpg>)

On the left, we have the unfolded (unrolled) RNN, while the representation on the right is the compressed form. A simplified mathematical representation of the RNN is as follows:

\\\[ \\begin{equation} h(t) = f( W\_h h(t-1) + W\_x x(t) + b) \\label{eq-rnn-vanilla} \\end{equation} \\\]

where \\(h(t)\\) represents the state of the unit at time \\(t\\), \\(x(t)\\) is the input at time \\(t\\).

> [!NOTE]
>
> **RNN and First-order Differential Equation**
>
> There are different views of the nature of time series data. Many of the time series datasets are generated by physical systems that follow the laws of physics. Mathematicians and physicists already studied and built up the theories of such systems and the framework we are looking into is dynamical systems.
>
> The vanilla RNN described in \\(\\eqref{eq-rnn-vanilla}\\) is quite similar to a first-order differential equation. For simplicity we use RELU for \\(f(\\cdot)\\).
>
> Note that
>
> \\\[ h(t - 1) = h(t) + \\sum\_{n=1}^{\\infty} \\frac{h^{(n)} (t)}{n!} (-1)^n, \\\]
>
> where \\(h^{(n)}(t)\\) is the \\(n\\)th derivative of \\(h(t)\\). Assuming that it converges and higher order doesn't contribute much, we rewrite \\(\\eqref{eq-rnn-vanilla}\\) as
>
> \\\[ \\begin{equation} \\frac{\\mathrm d h(t)}{\\mathrm d t} = x'(t) + W\_h h(t), \\label{eq-rnn-1st-order-diff-eq} \\end{equation} \\\]
>
> where
>
> \\\[ \\begin{align} x'(t) =&amp; W\_h^{-1} W\_x x(t) + b' \\\\ b' = &amp; W\_h^{-1} b. \\end{align} \\\]
>
> We have reduced the RNN formula to a first-order differential equation. Without discussing the details, we know that an exponential component \\(e^{W\_h t}\\) will rise in the solution. The component may explode or shrink.

Based on our intuition of differential equations, such a dynamical system usually just blows up or diminishes for a large number of iterations. It can also be shown explicitly if we write down the backprogation formula where the state in the far past contributes little to the gradient. This is the famous vanishing gradient problem in RNN[^/deep-learning-fundamentals/recurrent-neural-networks/#bib-Pascanu2012-qv]. One solution to this is to introduce memory in the iterations, e.g., long short-term memory (LSTM) [^/deep-learning-fundamentals/recurrent-neural-networks/#bib-Hochreiter1997-gm].

In the basic example of RNN shown above, the output of the hidden state is fed to itself in the next iteration. In theory, the value to feed back to the unit and how the input and output are calculated can be quite different in different setups [^/deep-learning-fundamentals/recurrent-neural-networks/#Amidi&Amidi][^/deep-learning-fundamentals/recurrent-neural-networks/#Karpathy2015]. In the section [Forecasting with RNN](<https://dl.leima.is/time-series-deep-learning/timeseries.rnn/index.md>), we will show some examples of the different setups.

## Step by Step: Unrolling an RNN

The same unit is applied at every time step. Step through four time steps to see the state \\(h(t)\\) being passed along.

A step-by-step diagram of a recurrent unit unrolled over four time steps. It needs JavaScript. In short: \\(h(t) = f(W\_h h(t-1) + W\_x x(t) + b)\\) is applied at each step with the same weights, so \\(h(4)\\) depends on all four inputs.

---

**Related:** [Forecasting with RNN](<https://dl.leima.is/time-series-deep-learning/timeseries.rnn/index.md>) · [Neural ODE](<https://dl.leima.is/dynamical-systems/neural-ode/index.md>)

[^/deep-learning-fundamentals/recurrent-neural-networks/#Amidi&Amidi]: Amidi A, Amidi S. CS 230. In: Recurrent Neural Networks Cheatsheet \[Internet\]. \[cited 22 Nov 2023\]. Available: [https://stanford.edu/\~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks](<https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks >)

[^/deep-learning-fundamentals/recurrent-neural-networks/#Karpathy2015]: Karpathy A. The Unreasonable Effectiveness of Recurrent Neural Networks. In: Andrej Karpathy blog \[Internet\]. 2015 \[cited 22 Nov 2023\]. Available: [https://karpathy.github.io/2015/05/21/rnn-effectiveness/](<https://karpathy.github.io/2015/05/21/rnn-effectiveness/ >)

[^/deep-learning-fundamentals/recurrent-neural-networks/#bib-Grossberg:2013]: Grossberg S.. Recurrent neural networks. *Scholarpedia*. 2013. [https://doi.org/10.4249/scholarpedia.1888](<https://doi.org/10.4249/scholarpedia.1888>)

[^/deep-learning-fundamentals/recurrent-neural-networks/#bib-Pascanu2012-qv]: Pascanu R., Mikolov T., Bengio Y.. On the difficulty of training Recurrent Neural Networks. 2012. [http://arxiv.org/abs/1211.5063](<http://arxiv.org/abs/1211.5063>)

[^/deep-learning-fundamentals/recurrent-neural-networks/#bib-Hochreiter1997-gm]: Hochreiter S., Schmidhuber J.. Long short-term memory. *Neural computation*. 1997. [http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735](<http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735>)
