# Time Series with Deep Learning Quick Bite > Time Series with Deep Learning Quick Bite A book on forecasting time series with deep learning, from the fundamentals of time series and neural networks to RNN, transformer, VAE, GAN, neural ODE and diffusion forecasters. Many pages come with a notebook; the notebooks are jupytext Python files under dl/notebooks/ in https://github.com/emptymalei/deep-learning. Every page below links to its Markdown version. ## Home - [Home]() ## Engineering Tips - [Coding Tips](): The engineering practices this book assumes, kept out of the main chapters: two references on writing good research code, and a checklist of tools for a Python deep learning project, covering the coding environment, code quality and the main packages. - [Python](): How to set up a Python project: pyenv for Python versions, poetry for dependencies and environments (with conda as the all-in-one alternative), pyproject.toml for the dependency list, pre-commit hooks such as black, isort and mypy for a consistent style, and docstrings and tests. ## Fundamentals of Time Series Forecasting - [Time Series Data and Statistical Forecasting Methods](): This chapter sets up the forecasting problem: use the last K steps of the targets and covariates, plus features known in advance, to predict the next H steps. It also shows how forecasting methods can be categorized. - [Terminologies of Time Series Data](): Two ideas from time series analysis that the rest of the book leans on: stationarity (the distribution of the series doesn't change over time, with a weaker form that only looks at the mean and autocorrelation) and serial dependence, measured by autocorrelation. It also notes how terminology differs between fields, such as panel data versus multivariate time series. - [Box-Cox Transformation](): Real series are often non-stationary, with a variance that changes with the level. The Box-Cox transformation, a family of transforms indexed by λ (λ = 0 is the log), reduces this; the air passenger data shows it in action. It doesn't always produce perfectly stationary data. - [Two-way Fixed Effects](): A two-way fixed effects model is a linear model for multivariate time series (panel data) with one effect per series and one per time step. A small constructed example with the prices and demand of three articles shows how it recovers their shared linear relation. - [Time Delayed Embedding](): Time delay embedding, also called rolling, rewrites a series as a table of fixed-length windows: each row holds p consecutive past values and the value to predict next. Neural networks need such fixed-length inputs, and Takens' theorem explains why the representation preserves the dynamics of the original series; the pendulum data illustrates it. - [DGP](): Time series come from many kinds of data generating processes: simple deterministic ones such as exponential growth and circular motion, pure noise, linear processes such as AR, and processes where several series interact. Simulating them builds intuition for modeling real data. - [DGP: Langevin Equation](): The Langevin equation describes Brownian motion: a velocity that decays exponentially while a random force keeps kicking it. Discretizing the equation gives a simple recipe for simulating a stochastic time series. - [Statistical Models](): A short tour of classical statistical forecasting models: ARIMA and how its components relate, simple exponential smoothing, which averages the latest observation with the previous forecast so that older history fades exponentially, and state space models. - [AR](): An autoregressive model predicts the next value from a weighted sum of previous values plus noise. Depending on its coefficients, an AR(1) process hovers around a constant, decays or grows exponentially; the AR(2) playground shows when a process stays stationary. - [VAR](): A vector autoregressive model extends AR to several series that influence each other: the next value of each series depends on the previous values of all of them. A VAR(1) blows up when the largest eigenvalue of its coefficient matrix exceeds 1 and stays stable otherwise. - [Synthetic Time Series](): Synthetic time series serve three purposes: benchmarking models on data whose dynamics are known, improving models through augmentation or transfer learning, and sharing data without exposing private or business information. Generating realistic series is hard, though generic generators such as GANs exist. - [Creating Synthetic Dataset](): How to generate synthetic series in code with the recipe-based generator in GluonTS, from random Gaussian noise to multivariate recipes. - [Augmentation](): Real datasets are often too small or too imbalanced to train deep models well. Augmentation creates extra training series by random transformations such as jittering, by mixing patterns from several series (for example DBA), or with generative models from AR to GANs. The page then covers how to use the synthetic data in training. - [Time Series Forecasting Tasks](): Forecasting tasks differ by how many series are involved. Univariate forecasting predicts the next H steps of one series from its last K steps; multivariate forecasting handles several series and their covariates, which may not be aligned at every time step. - [Naive Forecasts](): Simple baselines that every model should beat: repeat the last observation, forecast the mean of the past, or use simple exponential smoothing, which weights recent values more. Truncating the smoothing sum biases the forecast. - [Time Series Forecasting Evaluation](): Time series can't be split at random: the test data has to come after the training data, or information leaks from the future. Backtesting moves the split point forward in time, sliding or growing the training window, and evaluates on the block that follows, which gives several scores instead of one; a gap between train and test adds a safety margin. Shuffled cross-validation is rarely appropriate. - [Time Series Forecasting Metrics](): A metric turns forecast errors into one number, and the choice decides which model looks best. Scale-dependent errors (MAE, MSE, RMSE) are in the data's units, and the squared ones punish large misses most; percentage errors (MAPE, sMAPE) compare across series but blow up when the truth is near zero; scaled errors such as MASE compare against a naive forecast; probabilistic forecasts need scores such as the quantile loss or CRPS. - [CRPS](): CRPS scores a probabilistic forecast against the one value that actually happened. It compares the forecast's cumulative distribution with a step at the observed value: the closer and sharper the distribution around the truth, the lower the score. For a point forecast it reduces to the absolute error. - [Hierarchical Time Series Data](): Many datasets have structure: store sales add up to state sales, which add up to the total. A summing matrix S encodes these relations by mapping the bottom-level series to every level, and forecasts that respect it are called coherent. The M5 store sales serve as the example. - [Hierarchical Time Series Reconciliation](): Forecasting each level of a hierarchy separately gives numbers that don't add up. Reconciliation is a post-processing step: a matrix P maps the base forecasts to bottom-level forecasts, and the summing matrix S maps those back to all levels, which makes them coherent. MinT, one way to choose P, is worked through with examples, theory and real data. - [Time Series Datasets](): A list of real-world datasets used for forecasting (ECB exchange rates, NREL solar power, UCI electricity load, PeMS traffic) and of synthetic sources such as chaotic systems. - [Exchange Rate](): Euro foreign exchange reference rates from the ECB: 41 currencies starting on 1999-01-04, with example plots and a look at missing values. - [NREL Solar Power Data](): Synthetic solar power data from NREL for the United States in 2006, one file per plant in 5-minute steps. Nearly half of the records in the example file are zero, since there is no power at night. - [Electricity](): Electricity consumption of 370 clients from the UCI repository, in 15-minute steps from 2011 to the start of 2015 with no missing values, plus code to load and clean it. - [PeMS Traffic Data](): Traffic data from the California PeMS system and how to download it (registration required). No examples are shown yet. ## Trees - [Tree-based Models](): A decision tree predicts by asking a sequence of questions about the features. A toy example, deciding whether to work from home given health, weather and holidays, shows how a tree is described, how it is built from data and how easily it overfits. - [Random Forest](): A random forest is an ensemble of decision trees, each built on a random subset of the features, whose predictions are combined. Its predicting power is explained through the margin, the strength of the individual trees and the correlation between them; the page ends with the random forest regressor. - [Gradient Boosted Trees](): Boosted trees are also an ensemble, but an additive one: the prediction is the sum of the trees' predictions, and training adds one tree at a time, each chosen to reduce the remaining loss. LightGBM and XGBoost are the popular implementations. - [Forecasting with Trees](): Random forest and gradient boosted trees benchmarked on the air passenger data with Darts. Trees are poor at extrapolating beyond the data they were trained on, so the trend has to be removed first, without leaking information from the future; with that care, trees are simple, robust and strong forecasters. ## Fundamentals of Deep Learning - [Deep Learning Introduction](): An overview of the deep learning part: why a plain feedforward network is rarely enough, how self-supervised learning helps models generalize, and the notation used in the following chapters. - [Learning from Data](): Learning from data means extracting compressed knowledge about the world from data. Predictions can come from induction followed by deduction (first find a model, then apply it) or from transduction (predict directly from the data). - [Neural Networks](): An artificial neuron responds to its inputs more or less strongly, and a network composes many neurons into complicated functions. The page builds this intuition first, then explains why neural networks are universal approximators. - [Recurrent Neural Networks](): A recurrent neural network feeds the output of a unit back into itself, so its state at each time step depends on the current input and its previous state. Unrolled over time it is a chain of copies of the same unit, and it is closely related to a first-order differential equation. - [Convolutional Neural Networks]() - [Vanilla Transformers](): The transformer drops recurrence and relies on attention, which lets every position draw directly on every other position. The page summarizes the encoder-decoder structure, the attention mechanism and how positional encodings tell the model about order. - [Why Dynamical Systems](): Much time series data is generated by dynamical systems, such as the chaotic Lorenz system, and differential equations are the natural tool to model them. Finding clean, interpretable equations by hand is hard. - [Neural ODE](): A neural ODE combines neural networks with differential equations: a network models the rate of change of a system's state. Examples of differential equations build the intuition, and the finite difference form shows how such an equation is stepped forward in time. - [Introduction](): Energy-based models give every configuration of the variables an energy, a scalar from which a probability density follows through a partition function, as in statistical physics. Compatible configurations get low energy, that is, high probability density. - [Diffusion Models](): Diffusion models connect the data space and a latent space through a diffusion process that is assumed to be invertible, which keeps the otherwise intractable computations manageable. The page works through the objective, an example with five steps, the reverse process for a Gaussian and the optimization. - [Introduction](): The generative models in this part share three pieces: an encoder, an explicit latent space and a decoder. - [Autoregressive](): An autoregressive model describes each variable conditioned on the ones before it. The page also sorts out the different notations the literature uses for those preceding variables. - [AE](): An autoencoder encodes its input into a compact latent space and decodes it back. The simplest version is two small neural networks, an encoder and a decoder, trained with a loss such as cross entropy for binary data. - [VAE](): A variational autoencoder introduces a variational distribution q to handle the integral over the latent variable. Neural networks only have to model the parameters of simple Gaussian distributions, and the loss is explained term by term. - [Flow](): A flow model transforms a simple distribution into a complex one through a chain of coordinate transformations, tracking the density with the Jacobian. It runs in a generative direction (sample, then transform) or a normalizing direction (simplify the data distribution). - [GAN](): A GAN trains a generator to fool a discriminator, a minimax game whose optimum is reached only when the generated distribution equals the data distribution, compared through the Jensen-Shannon divergence. The page covers the vanilla GAN, f-GAN and InfoGAN, with code. ## Time Series Forecasting with Deep Learning - [Introduction](): Deep learning models train on batches, so a time series first has to be cut into fixed-length inputs and targets. The page shows the DataFrameDataset class of the ts_dl_utils package, which slides along the time axis with a history length, a horizon and an optional gap, and what its first data point looks like. - [Pendulum Dataset](): A synthetic dataset from a damped pendulum: the angle follows a cosine whose amplitude decays exponentially, with the period set by the length of the pendulum. The models in this chapter are trained to forecast this series, and the page gives the code that generates it. - [Forecasting with MLP](): A feedforward network with five hidden layers forecasts the pendulum series from 100 past steps: first one step ahead across a gap of 10 steps, then three steps ahead. It beats the naive forecast on all three metrics, and a small trained model runs in the browser at the end of the page. - [Forecasting with RNN](): An RNN with a single recurrent block and a hidden size of 64 forecasts the pendulum series, one step and three steps ahead, each across a gap of 10 steps. After a few seconds of training it follows the oscillation and scores much better than the naive forecast. - [Forecasting with Transformers](): An encoder-only transformer forecasts the univariate pendulum series: a linear layer shapes the input, the encoder processes it and a second linear layer produces the forecast. It beats the naive forecast one step and three steps ahead, and the page closes with the transformer variants designed for forecasting and the packages that implement them. - [Forecasting with CNN]() - [Forecasting with VAE]() - [Forecasting with Flow]() - [Forecasting with GAN]() - [Forecasting with Neural ODE](): A neural ODE, built with torchdyn and with a small neural network as its field, forecasts the pendulum series from 100 past steps. It suits this dataset because the pendulum is itself generated by a differential equation; the page reports one-step and three-step results against the naive forecast. - [Forecasting with Diffusion Models](): TimeGrad (Rasul et al., 2021) turns the denoising diffusion model into a probabilistic forecaster for multivariate series. The forecast is factorized autoregressively over time, an RNN state carries the time dynamics into the denoising network, and repeating the reverse process many times gives the quantiles. ## Supplementary - [About Supplementary Materials]() - [How to Run Our Notebooks](): How to run the notebooks of this book: install the Python environment with poetry, create a Jupyter kernel named deep-learning, and use the ts_dl_utils package, which holds the helpers shared by the tutorials. - [Box-Cox Transformation]() - [Pendulum Dataset]() - [Hierarchical Forecasting Using MinT]() - [Tree Basics]() - [Random Forest Basics]() - [Forecasting with Random Forest using Darts]() - [Forecasting with GBDT using Darts]() - [Creating Time Series Dataset (PyTorch)]() - [Forecasting with Feedforward Neural Networks]() - [Forecasting with RNN]() - [Forecasting with Transformer]() - [Forecasting with NeuralODE]() - [Generate Time Series Using Statistics]() - [Generate Time Series Using VAE]() - [Comparing Time Series]() - [Pretrained Models: TabPFN]() - [Diffusion Models]() - [Time Series Data Embedding Classical Methods]() - [Appendices]() - [Entropy](): Shannon entropy is the expected information content of a distribution. Cross entropy measures one distribution against another and splits into the entropy plus the KL divergence; it is widely used as the loss in classification. - [Mutual Information](): Mutual information of two random variables is the expected log ratio of their joint distribution to the product of their marginals. That makes it the KL divergence between the joint distribution and the product of the marginals. - [KL Divergence](): The Kullback-Leibler divergence compares two probability distributions. It is not symmetric, so it is not a proper distance; the page shows this with a Gaussian and a bimodal Gaussian mixture. It is a special case of the f-divergence. - [f-divergence](): The f-divergence compares two densities through a generating function f that is convex and satisfies f(1) = 0. Choosing f(x) = x log x gives the KL divergence. - [ELBO](): The evidence lower bound (ELBO) is a lower bound on the log probability of the data in a latent variable model. The page derives it twice: with Jensen's inequality, and from the KL divergence between an approximate posterior and the true posterior, where that KL divergence is the gap between the bound and the log probability. - [Alignment and Uniformity](): A good representation keeps similar instances close together and separates different ones. Wang and Isola (2020) call these two properties alignment and uniformity and describe both on a hypersphere. - [Gini Impurity](): Gini impurity measures how mixed the classes in a dataset are: it is 0 when every record has the same class and largest when the classes are evenly mixed. The page defines it, works through two small examples and shows its values for two and for three classes. - [Information Gain](): Information gain is the change in entropy when a dataset is split, and tree- based methods use it to judge a split. The page works through a decision tree node: its entropy before the split, the entropy of the two groups after it, and the gain as their difference. - [Generalization](): The generalization error is the difference between a model's loss on the whole population and its loss on the data it was fitted to. The population loss is unknown, so in practice it is approximated with a held-out test set. - [DTW](): Dynamic time warping (DTW) finds the best way to align two sequences and measures their misalignment with a distance defined recursively, much like the Levenshtein distance between two words. It is useful for comparing series of different lengths, for example a forecast and an actual series. - [DBA](): DTW barycenter averaging (DBA), proposed by Petitjean et al., averages a set of time series: it constructs the series that is their barycenter under the dynamic time warping distance. - [Introduction]() - [Deep Infomax](): Deep InfoMax trains an encoder by maximizing an estimate of the mutual information between the input and its encoding. Doing this for the whole input is often not enough, so it is also done between local patches of the feature maps and the encoding, which pushes the encoding to use information that is shared across the input. - [Contrastive Predictive Coding](): Contrastive predictive coding (CPC) encodes each segment of a sequence, models the encodings with an autoregressive model, and trains with the InfoNCE loss, which estimates a lower bound of mutual information. The encoders can then be used for downstream tasks such as classification. - [MADE]() - [MAF]() - [f-GAN](): f-GAN generalizes the GAN: instead of the Jensen-Shannon divergence, it estimates an f-divergence between the data distribution and the generated distribution. Training searches for the saddle point of a variational objective, and a particular choice of functions recovers the loss of the original GAN. - [InfoGAN](): InfoGAN makes the latent space of a GAN more interpretable. Next to the random noise it feeds the generator a latent code, and it adds the mutual information between that code and the generated data to the GAN objective, so the generator is penalized when it loses the information in the code. ## About - [Roadmap]() - [Changelog]() ## Other pages - [NCE]() - [Diffusion Model]() - [Diffusion Process]() - [LSTM]() - [CPC for Time Series Data]() - [CPC for Time Series Data]() - [RNN Dynamics During Inference]() - [Comparing RNN Models for Time Series Forecasting]() - [TimeVAE]() - [Utilities]() - [Transformer Explainer]() - [Transformer Forecaster with neuralforecast]() - [Transformer Forecaster with neuralforecast]() - [Transformer Forecaster with neuralforecast]() - [Transformer Forecaster with neuralforecast]() - [Transformer embeddings]() - [History of Transformer for Forecasting]() - [Adversarial Models]() - [Index]()