Skip to content

Training & Evaluation

Dataset

  • Simulation using pyroom acoustics or gpu-impulse-response packages
  • Kristina Tesch Code for pipeline examples (stationary & moving speakers)

Companion App

  • Create virtual test bed in Unity to quickly test and iterate on initial models (virtual agents -> talk to Ke Li)
  • Use pyroomacoustics/gpuRIR for audio simulation
    • VR room and audio simulation from same parameters
    • Challenges:
      • real-time HRTF with different head-angles
      • moving speakers
  • Start with desktop environment (cursor = viewing angle)
    • Iterate by adding more parameters (more speakers, noise, speaker variation, speaker angle & distance)
    • Move to VR at some point
    • Challenge: Multi-model input
      • lip movements on virtual agents not realistic
      • use videos of speakers?
  • Could be potential bachelor/master thesis
  • Visualize beamformer, target speaker confidence, ...?

User Study

  • Evaluate with students or medical patients as participants