Research Summary
A consolidated status summary of the PhD work: the CHAIS project (including the proposal's background and objectives), its current research plan, and — as one of the neuro-inspired architectures being explored within it — the work on oscillatory state-space models for speech enhancement.
CHAIS Project¶
[!note] CHAIS: Crossmodal Human-AI Collaboration for Selective Hearing Assistance Full proposal (PDF) — BMFTR pilot project on "Neurobiologically Inspired Artificial Intelligence" (Universität Hamburg / UKE), 549,288 € budget, 36-month funding period.
Background and Motivation¶
Humans can focus on a single speaker in noisy, multi-speaker environments (the cocktail party phenomenon) by exploiting the interaction between hierarchical cortical oscillations and auditory/visual signals — an ability current AI-based voice technologies cannot replicate. CHAIS is a cross-disciplinary cooperation between core AI disciplines (speech/multimodal signal processing, human-AI interaction) and neuroscience (neuroinformatics, computational neuroscience) to develop neuro-inspired crossmodal AI-based speech signal processing algorithms addressing the cocktail party problem, and to showcase their synergistic potential in a prototype smart glasses application for selective hearing assistance. In addition to the technical/algorithmic contribution, the project aims to demonstrate a two-way synergy: neuroscience findings improving AI, and AI models used to verify computational neuroscience hypotheses (e.g. the Global Neuronal Workspace).
Research objectives:
- O1 — Develop neuro-inspired crossmodal AI-based speech signal processing algorithms for the real-time Cocktail Party Problem.
- O2 — Improve and verify the understanding of the neural mechanisms of selective hearing.
- O3 — Design a prototype smart glasses application for selective hearing tasks, in a human-centered design process.
- O4 — Enhance usability, user experience, and performance of the glasses application and its algorithms.
Consortium¶
Four PIs across two University of Hamburg faculties (MIN and Medicine/UKE):
| PI | Institution | Role |
|---|---|---|
| Prof. Dr. Frank Steinicke | HCI, Dept. of Computer Science (MIN) | Coordinator; WP1 & WP4 leader — Human-AI Interaction |
| Prof. Dr.-Ing. Timo Gerkmann | Signal Processing, Dept. of Computer Science (MIN) | WP3 leader — speech and multimodal signal processing |
| Prof. Dr. Stefano Panzeri | Institute for Neural Information Processing (UKE) | WP2 leader — neuroinformatics, neuronal information processing |
| Prof. Dr. Claus C. Hilgetag | Institute of Computational Neuroscience (UKE) | WP5 leader — computational and network neuroscience |
Internally, this maps onto the three project partners tracked in the vault:
| Partner | Focus |
|---|---|
| Human-Computer Interaction | Develop human-usable assistive glasses, based on user needs and feedback. |
| Signal Processing | Improve AI-based real-time speech enhancement for hearing assistance in humans. |
| Neuronal Information Processing & Computational Neuroscience | Apply and gain insights into cognitive mechanisms in speech processing. |
Work Packages¶
| WP | Focus | Lead | Period |
|---|---|---|---|
| WP0 | Project coordination, documentation, dissemination | Steinicke | M1–36 |
| WP1 | Human-centered design process (requirement analysis, formative/summative evaluations) | Steinicke | M1–36 |
| WP2 | Neural architectures for speech processing (oscillation-based E/I neuron models, GNW-inspired network topology, topology×oscillation interaction) | Panzeri | M3–32 |
| WP3 | Crossmodal speech signal processing (translating WP2 findings into brain-inspired, distillable ANNs) | Gerkmann | M6–32 |
| WP4 | Prototype smart glasses application (companion app, gaze tracking, UI, multimodal feedback) | Steinicke | M12–36 |
| WP5 | Research ethics & data management | Hilgetag | M3–36 |
Key milestones (internal tracking):
| Month | Milestone |
|---|---|
| 6 | Requirement analysis (documentation and verification) |
| 12 | First models of neural architectures for speech processing (oscillatory and topology) |
| 12 | First evaluations of bio-inspired neural networks |
| 14 | Companion app / VR testbed |
| 18 | Early demonstrator of smart glasses |
| 24 | Neural architectures integrated in crossmodal speech signal processing |
| 30 | Summative evaluation of glasses |
| 36 | Final demonstrator and project finalization |
Full details: CHAIS Overview.
Current Research Plan¶
The plan proceeds in three phases, moving from a purely algorithmic entry point toward neural and then multi-modal speech enhancement, developed and evaluated using a purpose-built VR testbed.
Neural-inspired Speech Enhancement Within CHAIS's neuro-inspired architecture work (WP2/WP3, suggested by Timo, motivated by Stefano Panzeri's work), one concrete candidate architecture is an oscillatory state-space model (SSM). The question: does an entrained oscillatory inductive bias — phase and damping that are input-dependent rather than fixed — improve phase reconstruction in monaural speech enhancement, compared to non-oscillatory (real-decay) and non-selective (LTI) SSM baselines?
So far, a drop-in LTI oscillatory core (LinOSS) has not shown a clear advantage over Mamba or a complex selective SSM (Selective LRU) for speech enhancement. The open question is whether a genuinely selective oscillatory SSM — combining the harmonic-oscillator bias with input-dependent phase/damping — does better. As of September 2026, the stability question has been worked out in theory: IM-discretized selective LinOSS is unconditionally stable, and IMEX is stable above a damping floor (LinOSS Stability). Building on this, an entrained oscillatory S6 (a Mamba-S6 backbone with a complex scaled-rotation transition, see draft and S-LinOSS proposal) is designed as the next experiment.
Full details: Oscillatory SSM Overview.
Glasses - Phase 1 — Entrypoint, glasses with SE via beamformer Build a data-simulation/virtual testbed, decide on hardware integration (mic array, speakers, gyroscope, eye tracking, cameras) and algorithmic setup (delay/MVDR beamformer, Wiener filter). Step 1: MVP using head direction for beamforming. Step 2: integrate eye tracking for gaze-directed beamforming. - Phase 2 — Audio-only neural SE Move from algorithmic to neural speech enhancement, combining audio and view direction (building on SP lab work and bio-inspired approaches like LinOSS). - Phase 3 — Audio-visual fusion SE Add outward-facing cameras and audio-visual fusion (structured light vs. time-of-flight sensing), comparing mathematically-optimized vs. bio-inspired models.
[!Update] The Meta Project Aria has been requested by HCI group. We got confirmation that the request is being processed as of September 26. Tim Rolff is mainly dealing with the request and further inquiries by Meta.
CPP-Testbed Development and evaluation run on the CPP-Testbed, a VR application for studying the Cocktail Party Problem under controlled multi-speaker conditions (configurable speaker count/position/movement, room acoustics, and distractors), used to evaluate multi-channel, gaze-guided, and audio-visual speech enhancement, and to study psycho-acoustic auditory scene optimization. Data comes from acoustic simulation (pyroomacoustics / GPU impulse-response) with planned user studies involving students and, eventually, medical patients.
Full details: Research Plan, Assisted Listening Technology, Existing Research.