Psycho acoustic Auditory Scene Optimization
Psycho-acoustic Auditory Scene Optimization (PASO) describes the enhancement of a auditory scene in a Cocktail Party Problem scenario, based on psycho-acoustic insights into speech separation.
Selective Auditory Attention¶
SAA is the ability of humans to perceive an auditory target signal within a noisy, multi-talker environment.
Neural Mechanisms¶
The brain area involved in speech perception are the Posterior Surperior Temporal Gyrus, which is defined as the Lateral Parabelt Auditory Cortex.
(Mesgarani & Chang, 2012) examined the foundational mechanisms of auditory attention. They show that the neuronal spectro-temporal representation of speech shows a suppression of the masker speakers voice during attention tasks. The extend of suppression is time-dependent, increasing when a cue word identifies the target speaker in a mixture of speech. No neuronal patterns were identified in this study, suggesting higher-order processing to be responsible for the attentional masker suppression.
[!Paper Details]- Using electrode surface recordings in humans, the authors reconstruct the spectro-temporal representation (STR) of the perceived speech, encoded by the neuronal population of the listener. The subjects first listened to samples of a target speaker and a masker speaking on their own respectively. Then, they listened to a mixture of both speech signals, with task of attending to one a number-color-combination uttered by one of the speaker, based on a cue word. The authors then compare the reconstructed neuronal representation during the attention task with the single-speaker condition.
They find that when the subject identify the correct color-number combination, the STR of the mixture is significantly more correlated with the STR of the target speaker alone, compared to cases where subjects incorrectly identify the color-word combination.
The authors then examine the time dynamics of the auditory attention using a Attentional Modulation Index (AMI) which gives the composite score for the cross-correlation between the STR of mixed speech and target speaker (weighted positively) and the STR of mixed speech and masking speaker (weighted negatively). In other words, the score is positive if the correct speaker is attended to and negative, if the masker is attended to. They find that when subject correctly identify the number-color combination, the AMI is neutral/unbiased during the cue-word and then increased until the target utterance occurred later. In cases where subjects fail the task, the AMI tended to be negative and stayed negative during the target utterance, suggesting an attentional bias toward the masker during the cue word.
Lastly, the frequency-dependent activity of individual neurons is correlated with the STFT magnitude of the target and masker speaker. The find that the whole population shows increased correlation with the target speaker when the correct speaker was attended to, with no particular pattern, suggesting a higher-order processing inducing the attention-based masker suppression.
Later works by (Puschmann et al., 2024) examined the spatial selectivity along the cortical auditory hierarchy. It shows that low-level brain areas process both the target speaker and maskers speech to the same degree. But along the processing pathway, they find a gradual separation and suppression of the masker. In high-level brain areas, more related to semantic and linguistic processing (as examined by Mesgarani & Chang, 2012), they find barely to no residual informational content of the maskers speech.
[!Paper details]- The authors use fMRIs to record the subjects cortical activity when listening to a narrative A and narrative B. First they listen to a mixture of both narratives, being instructed to focus on narrative A. Then they listen to narrative A and B individually. After each run, subjects have to answer questions about narrative A and B.
For each brain region, a linear regression model is learned to estimate the brain activity when listening to the mixture of narratives as a linear combination of the brain activities of the individual narratives: \(y(t)=\beta_{A}x_{A}(t) + \beta_{B}x_{B}(t) + \epsilon\). If \(\beta_{A}\approx \beta_{B}\), then brain activity of listening to the mixture of speech is equally well estimated by the brain activies of listening to each individual speaker, suggesting both speakers are processed to similar extend. If however \(\beta_{A}>\beta_{B}\), then the brain activity of listening to the target speakers narrative is better estimating the brain activity of listening to the mixture of narratives, suggesting a suppression of the masker in the given brain area.
Using this regression method, the authors find similar \(\beta\) values for low-level brain areas and an increasing divergence (\(\beta_{A}<\beta_{B}\)) for increasing depth of corical processing. This suggests at early stages of auditory processing, both speakers are processed equally, whereas at higher stages, attentional modulation starts suppressing the masker, leaving almost only informational content of the target speaker at the highest stages, responsible for linguistic processing.
This work is support by (Hausfeld et al., 2024) , where fMRI was used to investigate the presence of target speaker and masker amplitude envelope in different brain areas. In higher auditory brain areas, the authors find no residual of the masker signal, and an inverse relationship between target speaker and the brain activity. They suggest a top-down control, where during low amplitude segments, the brain compensates with more neural activity. This is in line with
Methods¶
Volume Separation¶
The most common approach for target speaker enhancement is increasing the Signal-to-noise ratio, i.e. increasing the target speakers volume relative to other speakers and distractors. Two approaches increase the SNR, however they have different effects on speech ineligibility.
Pitch and Harmonic Structure¶
Spatial Separation¶
https://link.springer.com/chapter/10.1007/978-3-030-57100-9_8
Binaural Unmasking (BU)¶
Spatial speaker separation from binaural cues is a key for humans ability to separate speakers in low-SNR, multi-speaker scenarios. Humans can leverage differences in interaural parameters (ITD, IPD, ILD) between target signal and noise sources differs. This can be especially pronounced if the listener turns one ear toward the signal source, such that ITD and ILD are maximized due to relative ear orientation and head shadowing. Interestingly, the effect of unmasking due to ITD scales beyond naturally occuring ITD given the distance between ears, which might be exploited in multi-speaker speech enhancement scenarios.
Todo: More info on BLMD (binaural level masking difference) and amplitude fluctuations due to vector theory.
A crucial aspect that enables BU in humans is Phase Locking of action potentials. It enables the spatial differentiation in the early auditory cortical pathway. While phase locking breakes down for high-frequency spectral components, amplitude fluctuations due to interference effects might still enable the brain to leverage phase locking for these components.