Skip to content

Research Plan

Oscillatory SSM for Speech Enhancement

Suggested by Timo, due to relation to work by Stefano Panzeri. Refer to this Overview for more information.

Initial Steps

Phase 1: Entrypoint, Glasses with SE via Beamformer

  • Data simulation and virtual test-bed
  • Research material/experimental setup
    • Comparability to other papers with similar goals
    • UX vs experimental flexibility (tech should be quickly adaptable)
    • Question: Embedded/on-device vs. external computer
    • How to integrate
      • microphone array
      • speakers (in-ear vs in-frame)
      • gyroscope
      • eye tracking
      • cameras (RGB, infrared, depth, …?)
  • Research algorithmic setup
    • Delay Beamformer, MVDR Beamformer
    • Wiener Filter
    • What other signal input can be used?
  • Step 1: Build MVP, using head direction for beamforming
  • Step 2: Integrate eye tracking, using eye focus for beamforming

Phase 2: Audio-only Neural SE

  • Progress from purely algorithmic approaches to neural SE
  • Use audio + view direction for SE
    • SP lab research by Kristina Tesch
    • Bio-inspired approaches like LinOSS (UKE)
      • Status (September 2026): LTI LinOSS showed no clear advantage; now exploring selective oscillatory SSMs, see Oscillatory SSM project

Phase 3: Audio-Visual Fusion SE

  • Augment glasses with outward facing cameras
  • Use audio-visual fusion techniques for SE
  • Which modality in noisy environments?
    • Structured Light (e.g. RGB/infrared grid)
    • Time-of-Flight (lidar)
  • Which models?
    • Mathematical vs bio-inspired

CPP-Testbed

The CPP-Testbed is a VR application for studying the Cocktail Party Problem in a controlled multi-speaker environment. Applications include the evaluation of speech enhancement algorithms (multi-channel, gaze-guided, audio-visual), investigation of different environmental configurations (moving speakers, non-speech distractors, different spatial speaker positions), and Psycho-acoustic Auditory Scene Optimization.

  • Environment
    • Multi-speaker: Number, positions, sensible conversations vs babble
    • Target speaker selection: Head direction, gaze, pointing
    • realistic visual cues: virtual agents vs billboard agents
    • Differences of room setup: Open air, room shape, occluders
    • Distractors: Have street sounds, babble, etc. in background
  • Auralization
    • Realistic acoustic environment with HRTF, reverberation, defraction, etc.
  • Tasks:
    • Intelligibility vs naturalism (better understanding but unpleasant?)
    • Search problem: Space Party Rescue
    • Mixture of speech, cue word -> target word
      • All speakers utter similar sentences
      • One speaker utters cue word at some point, followed by a target utterance that is to be identified.
      • Works for binary speech attention, but does it work with more speakers and distractors and longer time-span?
      • Idea: Present visual indicator of cue word
        • All speakers speak at once, with one randomly uttering the cue word and then the target word
        • Repeat utterances, randomly choosing a speaker for the cue-word each time
        • Wait until subject identifies speaker with cue word and subsequent target words
        • Change cue word and repeat
        • Measure:
          • Time till correct identification
          • number of correct identifications
          • Listening effort
        • Pros:
          • Challenging, realistic auditory task
          • Requires continuous re-identification of target speaker
          • Search problem: Just fully removing all other speakers makes task more challenging than fractional attenuation.

Dataset & User Study

  • Simulation using pyroom acoustics or gpu-impulse-response packages
  • Kristina Tesch Code for pipeline examples (stationary & moving speakers)
  • User study: evaluate with students or medical patients as participants