HiPPO
Introduced in HiPPO: Recurrent Memory with Optimal Polynomial Projections (Gu et al., 2020).
The High-Order Polynomial Projection Operator (HiPPO) is a framework for online function approximation: at every time \(t\), the entire history of an input signal \(u(s),\ s\le t\) is compressed into a fixed-size vector of coefficients \(c(t)\in\mathbb{R}^{N}\). It is the basis for initializing the transition matrix \(A\) of the State Space Model and S4.
Idea¶
The history of \(u\) is projected onto a basis of \(N\) orthogonal polynomials with respect to a measure \(\mu^{(t)}\), which defines how much weight each point in the past receives. The coefficients \(c(t)\) of this projection are the optimal approximation of the history under \(\mu^{(t)}\). The key result is that, for suitable measures, these coefficients evolve according to a linear ODE:
This is exactly the form of a state-space model, with the coefficients \(c(t)\) as the hidden state. Discretizing the ODE (see Zero-Order Hold) gives a linear recurrence that updates the memory with every new input.
Measures¶
The choice of measure determines \(A\) and \(B\):
- LegT (translated Legendre): uniform weight over a sliding window \([t-\theta,t]\), i.e. memory of the recent past only.
- LagT (translated Laguerre): exponentially decaying weight over the past.
- LegS (scaled Legendre): uniform weight over the whole history \([0,t]\). The resulting dynamics \(\dot{c}(t)=-\frac{1}{t}Ac(t)+\frac{1}{t}Bu(t)\) are invariant to the time scale of the input.
For HiPPO-LegS, the matrices are
Role in SSMs¶
Randomly initialized recurrent matrices suffer from vanishing or exploding gradients over long sequences. The LSSL and S4 instead use the (time-invariant) HiPPO matrix as initialization for \(A\), which gives the state a principled long-range memory. Diagonal variants (S4D) and later models like Mamba keep HiPPO-inspired initializations. LinOSS is motivated by replacing this strict parameterization with a simpler one that is stable by construction.