Skip to content

CPP Testbed

The CPP-Testbed is a VR application for studying the Cocktail Party Problem in a controlled multi-speaker environment. Applications include the evaluation of speech enhancement algorithms (multi-channel, gaze-guided, audio-visual), investigation of different environmental configurations (moving speakers, non-speech distractors, different spatial speaker positions), and Psycho-acoustic Auditory Scene Optimization.

  • Environment
    • Multi-speaker: Number, positions, sensible conversations vs babble
    • Target speaker selection: Head direction, gaze, pointing
    • realistic visual cues: virtual agents vs billboard agents
    • Differences of room setup: Open air, room shape, occluders
    • Distractors: Have street sounds, babble, etc. in background
  • Auralization
    • Realistic acoustic environment with HRTF, reverberation, defraction, etc.
  • Tasks:
    • Intelligibility vs naturalism (better understanding but unpleasant?)
    • Search problem: Space Party Rescue
    • Mixture of speech, cue word -> target word
      • All speakers utter similar sentences
      • One speaker utters cue word at some point, followed by a target utterance that is to be identified.
      • Works for binary speech attention, but does it work with more speakers and distractors and longer time-span?
      • Idea: Present visual indicator of cue word
        • All speakers speak at once, with one randomly uttering the cue word and then the target word
        • Repeat utterances, randomly choosing a speaker for the cue-word each time
        • Wait until subject identifies speaker with cue word and subsequent target words
        • Change cue word and repeat
        • Measure:
          • Time till correct identification
          • number of correct identifications
          • Listening effort
        • Pros:
          • Challenging, realistic auditory task
          • Requires continuous re-identification of target speaker
          • Search problem: Just fully removing all other speakers makes task more challenging than fractional attenuation.

Todo

  • Create virtual test bed in Unity to quickly test and iterate on initial models (virtual agents -> talk to Ke Li)
  • Use pyroomacoustics/gpuRIR for audio simulation
    • VR room and audio simulation from same parameters
    • Challenges:
      • real-time HRTF with different head-angles
      • moving speakers
    • Better: Microsoft Acoustic unity
  • Start with desktop environment (cursor = viewing angle)
    • Iterate by adding more parameters (more speakers, noise, speaker variation, speaker angle & distance)
    • Move to VR at some point
    • Challenge: Multi-model input
      • lip movements on virtual agents not realistic
      • use videos of speakers?
  • Could be potential bachelor/master thesis
  • Visualize beamformer, target speaker confidence, …?