Skip to content

Autoregressive Model

Generative AR basics

An autoregressive (AR) model is autoregressive,

\[ \begin{equation} \log p_\theta (x) = \sum_{t=1}^T \log p_\theta ( x_{t} \mid \{x_{<t}\} ). \end{equation} \]
Notations and Conventions

In AR models, we have to mention the preceding nodes (\(\{x_{<t}\}\)) of a specific node (\(x_{t}\)). For \(t=5\), the relations between \(\{x_{<5}\}\) and \(x_5\) are shown in the following illustration.

There are different notations for such relations.

  • In Uria et al., the authors use \(p(x_{o_d}\mid \mathbf x_{o_{<d}})\) 1.
  • In Liu et al. and Papamakarios et al., the authors use \(p(x_{t}\mid \mathbf x_{1:t-1})\) 64.
  • In Germain et al., the authors use \(p(x_t\mid \mathbf x_{<t})\) 5.

In the current review, we expanded the vector notation \(\mathbf x_{<t}\) into a set notation as it is not necessarily a vector.

  1. Uria B, Côté M-A, Gregor K, Murray I, Larochelle H. Neural Autoregressive Distribution Estimation. arXiv [cs.LG]. 2016. Available: 

  2. Triebe O, Laptev N, Rajagopal R. AR-Net: A simple Auto-Regressive Neural Network for time-series. arXiv [cs.LG]. 2019. Available: 

  3. Ho G. George Ho. In: Eigenfoo [Internet]. 9 Mar 2019 [cited 19 Sep 2021]. Available: 

  4. Papamakarios G, Pavlakou T, Murray I. Masked Autoregressive Flow for Density Estimation. arXiv [stat.ML]. 2017. Available: 

  5. Germain M, Gregor K, Murray I, Larochelle H. MADE: Masked autoencoder for distribution estimation. 32nd International Conference on Machine Learning, ICML 2015. 2015;2: 881–889. Available: 

  6. Liu X, Zhang F, Hou Z, Wang Z, Mian L, Zhang J, et al. Self-supervised Learning: Generative or Contrastive. arXiv [cs.LG]. 2020. Available: 

  7. Lippe P. Tutorial 12: Autoregressive Image Modeling — UvA DL Notebooks v1.1 documentation. In: UvA Deep Learning Tutorials [Internet]. [cited 20 Sep 2021]. Available: 

  8. rogen-george. rogen-george/Deep-Autoregressive-Model. In: GitHub [Internet]. [cited 20 Sep 2021]. Available: 

Contributors: LM