Skip to content

Spiking Neural Networks

Spiking neural networks (SNNs) are neural networks whose neurons communicate through discrete spikes over time, like biological neurons (see Neural Spiking), instead of continuous activations. They are often called the third generation of neural network models (Maass, 1997). Since information is carried by when neurons fire, SNNs are naturally suited for temporal signals and bio-inspired mechanisms like oscillations and Phase Locking.

Leaky Integrate-and-Fire Neuron

The most common neuron model is the leaky integrate-and-fire (LIF) neuron. Its membrane potential \(V\) integrates the input current \(I(t)\) and leaks back to a resting potential:

\[ \tau_{m}\frac{dV}{dt}=-(V-V_{\text{rest}})+RI(t) \]

When \(V\) reaches the threshold \(V_{\text{th}}\), the neuron emits a spike and \(V\) is reset. In discrete time, with decay factor \(\beta\), input weights \(W\) and reset by subtraction, this becomes

\[ \begin{align} V_{t} &= \beta V_{t-1}+Wx_{t}-S_{t-1}V_{\text{th}}\\ S_{t} &= \Theta(V_{t}-V_{\text{th}}) \end{align} \]

where \(\Theta\) is the Heaviside step function and \(S_{t}\in\{0,1\}\) the output spike. Without the threshold, the membrane is a first-order linear recurrence with real decay, comparable to a real-valued diagonal State Space Model.

Spike Encoding

Continuous inputs must be encoded as spike trains:

  • Rate coding: input magnitude is encoded as firing frequency.
  • Temporal (latency) coding: larger inputs fire earlier.
  • Phase coding: information is encoded in the spike time relative to an ongoing oscillation. Auditory information and speech are encoded this way in the brain.

Training

The spike function \(\Theta\) has zero gradient almost everywhere, so standard [[Backpropagation]] does not work directly. Common approaches are:

  • Surrogate gradients: use \(\Theta\) in the forward pass, but a smooth approximation of its derivative in the backward pass (Neftci et al., 2019).
  • ANN-to-SNN conversion: train a conventional network and map its activations to firing rates, usually with only minor performance loss.
  • Local learning rules: biologically plausible, unsupervised rules like spike-timing-dependent plasticity (STDP).

Strengths and Weaknesses

SNNs are sparse and event-driven, so they can be very energy efficient. These benefits are only realized on neuromorphic hardware, however; on GPUs, SNNs have to be simulated step by step. Their use for speech enhancement, including the spiking BOSSA model, is discussed in Existing Research.