AutoBNN: Probabilistic Forecasting of Time Series Using Compositional Bayesian Neural Networks
AutoBNN is grounded in research developed over the last decade, which has enhanced predictive accuracy by applying time series modeling with Gaussian processes (GPs) that feature learned kernel structures. The kernel function of a GP encapsulates assumptions regarding the modeled function, such as trends, periodicity, or noise. With learned GP kernels, the kernel function is compositionally defined: it consists of either a base kernel (including Linear, Quadratic, Periodic, Matérn, or ExponentiatedQuadratic) or a composite formed by combining multiple kernel functions using operators like Addition, Multiplication, or ChangePoint. This compositional kernel framework serves two essential purposes. First, it is straightforward enough for users who possess expertise in their data, but may not have in-depth knowledge of GPs, allowing them to create a reasonable prior for their time series. Second, methods such as Sequential Monte Carlo can facilitate discrete searches over smaller structures and yield interpretable results.
AutoBNN enhances these concepts by substituting GPs with Bayesian neural networks (BNNs) while preserving the compositional kernel structure. A BNN is characterized by having a probability distribution over its weights rather than a static set of weights, which leads to a distribution over outputs and effectively captures uncertainty in predictions. BNNs offer several advantages compared to GPs: First, training large GPs often requires significant computational resources, with traditional training algorithms scaling cubically with the number of data points in the time series. In contrast, for a fixed width, training a BNN is often approximately linear in relation to the number of data points. Second, BNNs are better suited for hardware acceleration on GPUs and TPUs than the operations involved in GP training. Third, compositional BNNs can be seamlessly integrated with traditional deep BNNs, which can learn features autonomously. One potential approach is creating “hybrid” architectures, where users define a top-level structure, such as Add(Linear, Periodic, Deep), while the deep BNN is responsible for learning the contributions from high-dimensional covariate information.
How can one transform a GP with compositional kernels into a BNN? Research indicates that a single-layer neural network generally converges to a GP as the number of neurons (or “width”) approaches infinity. More recently, many well-known GP kernels, including Matérn, Exponentiated Quadratic, Polynomial, or Periodic, have been demonstrated to correspond to infinite-width BNNs when appropriate activation functions and weight distributions are chosen. Moreover, these BNNs remain closely aligned with their corresponding GPs, even when the width is significantly less than infinite. For instance, the figures below illustrate the differences in covariance between pairs of observations and regression outcomes from the true GPs compared to their corresponding width-10 neural network representations.
