Ideas
SSM-based Architectures¶
Oscillatory SSMs for Phase-Aware SE¶
Details: Entrained Oscillatory State-Space Models for Phase-Aware Speech Enhancement
Complex SSMs excel in frequency-based domains like vision and audio (see Mamba 3, p16). Also we observe cortical oscillations to play a critical part in auditory cortical processing. Since SEMamba, Mamba is the de-facto model to be incorporated for any kind of speech processing task. This invites the question of whether oscillatory dynamics might improve speech enhancement in comparison to other non-selective and selective SSM baselines.
- LinOSS improves upon S4D, but not upon Mamba
- Problem: Incorporating selectivity into LinOSS non-trivial.
- Finding: Incorporating complex state-space with data-controlled magnitude and phase improves phase-tracking. But complex state-space only introduces rotary state-space, not genuine oscillatory inductive bias.
- See Results - June 26
Schrödinger Bridge U-Net Mamba¶
SBMamba shows that single step generative inference is highly feasible with selective SSMs like Mamba. MambAttention proves that interleaving Mamba and Attention layers improves performance and generalization. RWSA-MambaUNet incorporates these hybrid layers into a U-Net architecture to achieve SOTA performance on the URGENT 2026 dataset. The ideas neatly combine: Use RWSA-MambaUNet as the backbone for Schrödinger Bridge training.
- Potentially even incorporate the cross-band blocks from SBMamba/SpatialNet for multi-channel SE.
- If Selective LRU proves successful, also check its incorporation instead of Mamba.
- Same UNet architecture could be incorporated into SGMSE architecture?
Predictive Coding¶
Neural Oscillation and [[Global Neuronal Workspace (GNW)]] are basic mechanisms in the brain, however incorporating them in signal processing algorithms just because they exist in the brain is ill-posed. [[Predictive Coding]] on the other hand is a powerful neuro-computational framework embedded in the [[Bayesian Probability]] theory. Bayesian approaches are fundamental to statistical signal processing and also have been well-suited model for human psychophysical behavior. Thus, predictive coding may yield insights for improving the design of deep neural networks.
Predictive coding poses that the human brain is constantly generating and updating a mental model of the environment from prior beliefs, while sensory observations are constantly compared to this model. The brain then tries to minimize the error between mental model, informed top-down from prior beliefs, and observation, informed by bottom-up sensory signals. The difference between model and observation is called "free-energy".
Precision Weighting¶
https://arxiv.org/pdf/2405.02384 https://pmc.ncbi.nlm.nih.gov/articles/PMC3001758/