SSMs for Speech Enhancement

This is a short review of the frontier of SE models in June of 2026. The SEMamba model was published in 2024, which introduces Mamba as a backbone in MP-SENet, and brought SSMs to the forefront of speech processing. The model placed second at the URGENT 2024 challenge. Several models build on top of SEMamba:

  • BiMamba Analysis paper showing that bidirectional Mamba with separate B and C performs best for speech enhancement tasks.
  • MC-SEMamba Simple multi-channel adaption of SEMamba. SEMamba's first convolutional layer has two input channels, one each for magnitude and phase; for MC-SEMamba this is simply scaled by the number of input channels. The output stays single-channel: magnitude mask and phase are predicted for one reference microphone.
  • USEMamba Iteration on SEMamba that simplifies the losses and makes the model a hybrid of generative and discriminative, improving on tasks like deverberation and bandwidth extension.
  • AVSEMamba Audio-visual version of SEMamba. A pretrained ResNet-18 extracts spatio-temporal features from full-face video and stacks them with the magnitude and phase as an extra input channel to SEMamba.
  • SEMamba++ Uses Mamba only for the time axis; for the frequency axis, blocks based on [[Fourier Analysis Networks]] better adapt to the periodic nature of the data. Multi-timescale processing is added to better capture global speech features. The adapted frequency blocks generalize better, and the model outperforms USEMamba and other SOTA SSMs.
  • MambAttention Based on the observation that hybrid Mamba/Multihead-Attention approaches improve performance and generalization, this model precedes Mamba with a Transformer pass, improving out-of-domain performance significantly.
  • RWSA-MambaUNet Follow-up to MambAttention that incorporates the hybrid blocks into a U-Net architecture. The down- and upsampling improves performance and generalization.
  • oSpatialNet-Mamba Incorporates Mamba into SpatialNet and uses causal convolutions for online multi-channel streaming. Mamba outperforms retention networks and MSA attention here.
  • SBMamba Incorporates the oSpatialNet architecture into a Schrödinger Bridge optimal-transport regime. The authors propose SSMs as a learned optimal-control solution to the Markov-style Schrödinger System. SBMamba achieves SOTA streaming performance at very low parameter counts with a single inference step.

An alternative branch of SSM-based SE is LTI-SSMs for low-latency speech processing, introduced by SaShiMi in 2022. SaShiMi works on the raw waveform and uses an hourglass down- and upsampling architecture, similar to U-Net but using simple reshape + linear projections instead of convolutions. The upsampling path is interleaved with S4 blocks. The model outperformed WaveNet with very large context windows.

  • S4ND U-Net An iteration on SaShiMi, replacing S4 with S4ND layers. S4ND runs the SISO S4 model over multiple channels in parallel, achieving Mamba-1-style MIMO capabilities. With much fewer parameters, S4ND matched and outperformed other SOTA time-domain models; the authors also found the architecture worked better in the time domain than the time-frequency domain.
  • aTTENuate Another SaShiMi variation, but with a U-Net architecture and more SSMs, including non-linearities and pre-convolutions. Achieves very low latency and decent PESQ on VoiceBank-DEMAND.
  • TRAMBA Transformer-Mamba hybrid for time-domain speech enhancement.

The core models of this lineage are described in their own notes: MP-SENet, SEMamba and USEMamba. The underlying SSM layers are collected in State-Space Models.