Assisted Listening Technology
The technical and research methodology for CHAIS: candidate hardware, the surrounding technology landscape, the VR testbed used to study the Cocktail Party Problem, and the phased research plan.
Research Glasses¶
Overview on auditive assistive glasses for solving Cocktail Party Problem. Potentially suited glasses for development are…
| Glasses | Notes |
|---|---|
| Meta Project Aria | optimal candidate? |
| Magic Leap | |
| Zinn Labs DK1 Evaluation Kit | doesn't seem currently available |
| Pupil Labs Neon | no speakers |
| BrillianLabs Halo | no eye tracking |
| Tobii Pro Glasses | single mic, no speakers |
| Apple Vision Pro | requires Apple ecosystem |
| Hololense | eye tracking? |
| Quest Pro | only "see-through" mode, but probably easiest to develop |
| Meta Ray-Ban | limited access through Meta account, latency due to Bluetooth streaming |
[!Update] The Meta Project Aria has been requested by HCI group. We got confirmation that the request is being processed as of September 26. Tim Rolff is mainly dealing with the request and further inquiries by Meta.
Existing Technology¶
Current products and research into multi-speaker speech enhancement are focused on learning-based approaches. Gaze-directed beamforming has been explored, but is limited to traditional signal processing approaches. An alternative to gaze fixation, called Auditory Attention Decoding, uses EEG/EOG signals to identify active speakers. Neuro-inspired approaches mostly explore network topology and Spiking Neural Networks. Audio-visual fusion techniques have been shown to increase robustness of SE in multi-speaker scenarios. The efficacy of assistive hearing devices have been evaluated by rating listening effort using questionnaires, but also via electrical signals and oxygenation in the brain.
Commercial Products¶
Commercial hearing-assistive products fall into several categories
Assistive Glasses¶
Hearing-assistive glasses have the advantage of supporting multi-modal input and output and more spaced out microphone arrays. They typically fall into three categories: - Glasses that enhance the speaker, utilizing the viewing direction. - Glasses that transcribe the speaker, projecting captions of the speech into the lenses. - Glasses that are multi-media consumption devices. They may support both speaker enhancement and transcription, but often with higher latency.
Nuance Audio Glasses¶
Source: https://www.nuanceaudio.com
Uses directional microphones, beamforming and noise suppression to improve speech comprehension in cocktail party scenarios (office, conferences, parties, restaurants). Uses head-direction for speech enhancement. Has in-frame open-ear speakers and dual front-facing microphones.
Glasses can be configured via… - remote control - on-frame touch button - phone app
Configuration options are noise suppression level, volume, and mode (360 vs frontal), plus a set of presets:
| Preset | Description |
|---|---|
| A | Flat mild hearing loss. Slightly amplifies sounds across low to high pitches |
| B | Sloping mild hearing loss. Amplification in the higher pitches |
| C | Sloping moderate hearing loss. Amplification in the higher pitches with increased volume |
| D | Flat moderate hearing loss. Slightly amplifies sounds across low to high pitches with increased volume |
Has user calibration: Glasses play short bursts of sound, then user reads pre-configured sentences aloud. Target is to reduce acoustic feedback. Probably some approximation of [[Head-related Transfer Function (HRTF)]]. The glasses also allow for tracking of noise level via app, so noise suppression level can be adjusted accordingly (manual adjustment). Works for people with mild to severe hearing loss.
Might serve as a baseline of commercially available products.
Meta Ray-Ban Glasses¶
Source: - https://www.meta.com/de/ai-glasses/ray-ban-meta/ - https://www.ray-ban.com/germany/electronics/RW4012ray-ban%20meta%20wayfarer%20-%20gen%202-grau%20transparent/8056262909898
Smart glasses with main goal of facilitating low-threshold media and AI interaction. Has array with 5 microphones (2 in each temple, 1 in nose-bridge) and directional Bluetooth open-ear speakers with adaptive volume of up to 76dB. Uses dual (?) cameras with 12MP ultrawide. Has single-eye display via Waveguides.
Can be used for - real-time translation (of speech and text in environment) - real-time STT - speech enhancement - via head direction - makes speech of focused person brighter - Media and AI interaction - Capturing images and videos - Social media - Be-my-eyes: Voluntary helpers guide blind persons via camera output
Interaction via - Touchpad in frame - Uses gesture control - Adjust volume, standby, navigation - Dedicated recording button - Phone app - Neural Band for gesture control via EMG
Bluetooth adds latency and privacy concerns, selling point of #Xander Glasses is on-device processing. Meta seems to focus on speaker enhancement using Conversation Focus.
OrCam Hear¶
Source: https://www.red-dot.org/de/project/orcam-hear-53370
An alternative to using glasses for multi-modal input: A necklace with integrated camera and chip to perform speaker identification and speech enhancement.
Speech Transcription Glasses¶
Xander Glasses¶
Source: https://www.xanderglasses.com/xanderglasses
Glasses allow for real-time STT, displayed via binocular AR display based on Vuzix. No app connection, allowing for low-latency, real-time captioning.
Uses dual microphones with noise suppression for captioning. Claims 90% captioning accuracy. No interaction possibilities except turning it on and off. Works for people with mild to complete hearing loss.
Competes with #Meta Ray-Ban Glasses, selling point is low-latency and privacy-preserving on-device processing.
Xrai Glasses¶
Source: https://xrai.glass/
Very similar to #Xander Glasses, claim 98% transcription accuracy, but also uses Bluetooth connection for full performance.
EvenRealities G2¶
Source: https://www.evenrealities.com/de-DE/smart-glasses
Glasses with transcription, but also translation and media consumption. Uses Waveguide display like Meta Ray-Ban glasses.
Hearing Aids¶
Commercial hearing aids use a mix of classical signal processing and AI-based models. Current focus are multi-speaker scenarios, multi-directionality and environment adaption.
Signia Speech Enhancement¶
Source: https://www.signia-pro.com
No products of its own, but a technology provider. Provides a whitepaper for a multi-beamformer, which should improve speech enhancement in multi-speaker scenarios. Uses phase-inversion techniques. Idea is to apply gain in speaker directions and suppression anywhere else. No "focused speaker" scenario.
Phonak AutoSense¶
Source: - https://www.phonak.com/de-de/hoerloesungen/hoersysteme/audeo-sphere - https://www.phonak.com/en-int/professionals/innovations
Speech enhancement in multi-speaker scenarios. "Direction-independent" speech enhancement likely means an ML model trained to remove background babble, running on a dedicated chip.
Starkey¶
Source: https://www.starkeypro.com/government/continue-learning/research-and-publications
Starkey produces AI-based speaker enhancement software, that promises improvements in multi-directional speaker sources.
Fortell¶
Source: - https://www.fortell.com/how-it-works - https://cdn.prod.website-files.com/68483ae9b9193f5347f397e0/69a1b1a1be15067025217cf0_e06ee42832504254d65e9f3f9bcf0304_Spatial%20AI%20Improves%20Speech%20Intelligibility%20for%20Hearing%20Aid%20Wearers.pdf
Recent competitor, uses high-performance TSMC chip and AI-based source separation together with noise suppression and speaker emphasis. Promises magnitude improvement in multi-speaker scenarios, even when the main speaker is quieter than the noise.
Over-the-Counter Earbuds¶
These products are cheaper (<1000$) and mostly prescription-free alternatives to hearing aids.
- Linner Earbuds Have 16 WDRC channels (wide dynamic range compression) that allow for separate processing and gains of different frequency bands.
- Sony CRE-E10 Uses #Signia Speech Enhancement to improve speaker quality