Time Series Forecasting using Deep Learning

Forecasting the future is an extremely valuable superpower. The forecasting game has been dominated by statisticians who are real experts in time series problems. As the amount of data increases, many of the statistical methods are not squeezing enough out of the massive datasets. Consequently, time series forecasting using deep learning emerges and became a fast-growing field. It is trendy, not only in LinkedIn debates but also in academic papers. We plotted the number of related publications per year using the keyword "deep learning forecasting" on dimensions.ai) ².

This chart is obtained on 2022-08-06, from Digital Science’s Dimensions platform, available at https://app.dimensions.ai

On the other hand, deep learning methods are not yet winning all the games of forecasting. Time series forecasting is a complicated problem with a great variety of data generating processes (DGP). Some argue that we don't need deep learning to forecast since well tuned statistical models and trees are already performing well and are faster and more interpretable than deep neural networks³⁴. Ensembles of statistical models performing great, even outperforming many deep learning models on the M3 data¹.

However, deep learning models are picking up speed. In the M5 competition, deep learning "have shown forecasting potential, motivating further research in this direction"⁵. As the complexity and size of time series data are growing and more and more deep learning forecasting models are being developed, forecasting with deep learning is on the path to be an important alternative to statistical forecasting methods.

In Coding Tips, we provide coding tips to help some readers set up the development environment. In Deep Learning Fundamentals, we introduce the fundamentals of deep neural networks and their practices. For completeness, we also provide code and derivations for the models. With these two parts, we introduce time series data and statistical forecasting models in Time Series Forecasting Fundamentals, where we discuss methods to analyze time series data, several universal data generating processes of time series data, and some statistical forecasting methods. Finally, we fulfill our promise in the title in Time Series Forecasting with Deep Learning.

Blueprint

The following is my first version of the blueprint.

• Engineering Tips
  • Environment, VSCode, Git, ...
  • Python Project Tips
• Fundamentals of Time Series Forecasting
  • Time Series Data and Terminologies
  • Transformation of Time Series
  • Two-way Fixed Effects
  • Time Delayed Embedding
  • Data Generating Process (DGP)
  • DGP: Langevin Equation
  • Kindergarten Models for Time Series Forecasting
    • Statistical Models
    • Statistical Model: AR
    • Statistical Model: VAR
  • Synthetic Datasets
    • Synthetic Time Series
    • Creating Synthetic Dataset
  • Data Augmentation
  • Forecasting
    • Time Series Forecasting Tasks
    • Naive Forecasts
  • Evaluation and Metrics
    • Time Series Forecasting Evaluation
    • Time Series Forecasting Metrics
      • CRPS
  • Hierarchical Time Series
    • Hierarchical Time Series Data
    • Hierarchical Time Series Reconciliation
  • Some Useful Datasets
• Trees
  • Tree-based Models
  • Random Forest
  • Gradient Boosted Trees
  • Forecasting with Trees
• Fundamentals of Deep Learning
  • Deep Learning Introduction
  • Learning from Data
  • Neural Networks
  • Recurrent Neural Networks
  • Convolutional Neural Networks
  • Transformers
  • Dynamical Systems
    • Why Dynamical Systems
    • Neural ODE
  • Energy-based Models
    • Diffusion Models
  • Generative Models
    • Autoregressive Model
    • Auto-Encoder
    • Variational Auto-Encoder
    • Flow
    • Generative Adversarial Network (GAN)
• Time Series Forecasting with Deep Learning
  • A Few Datasets
  • Forecasting with MLP
  • Forecasting with RNN
  • Forecasting with Transformers
    • TFT
    • DLinear
    • NLinear
  • Forecasting with CNN
  • Forecasting with VAE
  • Forecasting with Flow
  • Forecasting with GAN
  • Forecasting with Neural ODE
  • Forecasting with Diffusion Models
• Extras Topics, Supplementary Concepts, and Code
  • DTW and DBA
  • f-GAN
  • Info-GAN
  • Spatial-temporal Models, e.g., GNN
  • Conformal Prediction
  • Graph Neural Networks
  • Spiking Neural Networks
  • Deep Infomax
  • Contrastive Predictive Coding
  • MADE
  • MAF
  • ...

---

1. Nixtla. statsforecast/experiments/m3 at main · Nixtla/statsforecast. In: GitHub [Internet]. [cited 12 Dec 2022]. Available: https://github.com/Nixtla/statsforecast/tree/main/experiments/m3 ↩

2. Hook DW, Porter SJ, Herzog C. Dimensions: Building context for search and evaluation. Frontiers in Research Metrics and Analytics 2018; 3: 23. ↩

3. Elsayed S, Thyssens D, Rashed A, Jomaa HS, Schmidt-Thieme L. Do we really need deep learning models for time series forecasting? 2021. doi:10.48550/ARXIV.2101.02118. ↩

4. Grinsztajn L, Oyallon E, Varoquaux G. Why do tree-based models still outperform deep learning on tabular data? 2022. doi:10.48550/ARXIV.2207.08815. ↩

5. Makridakis S, Spiliotis E, Assimakopoulos V. M5 accuracy competition: Results, findings, and conclusions. International journal of forecasting 2022; 38: 1346–1364. ↩

---

Contributors: LM