LinOSS
Introduced in Oscillatory State-Space Models.
The Linear Oscillatory State-Space model (LinOSS) introduces forced linear second-order ODEs (Differential Equation#Ordinary Differential Equation (ODE)), inspired by the oscillatory dynamics of cortical neurons. Compared to the ordinary State Space Model, it achieves more stable dynamics and better models long-range interactions with a simpler and less restrictive parameterization.
Idea¶
The State Space Model (SSM) is a dynamic system, meaning its recurrent state (continuous memory) is evolving over time. A common issue in these models are vanishing or exploding gradients, where this internal states values explode or vanish in size. The ordinary SSM uses the strict [[HiPPO]] parameterization to initialize the transition matrix, which guarantee bounded growth over time. Finding less strict parameterizations allows for more easily trainable and more expressive models. The LinOSS models the inner states transitions using insights from linear oscillations, which allows for very simple parameterizations of the transition matrix (non-negative diagonal matrix) to guarantee stable inner state dynamics.
Oscillatory State Space¶
At the core of LinOSS is in how the inner states dynamics are modeled over time. The ordinary SSM uses the first-order ODE \(y^{\prime}(t)=Ay(t)+Bu(t)\) to model the dynamics of the state space \(y(t)\). In contrast, LinOSS uses the second-order ODE
to model the dynamics. This is the functional form of a Harmonic Oscillator#Forced Harmonic Oscillator, an approximation of neural oscillation. Neural oscillations represent synchronizations in neural spike trains (Neural Spiking#Oscillations) and are the foundation of many neural functions like information processing, perception, temporal decoding, memory, sleep and more.