Skip to content

SFG Mamba

Summary

The paper proposes a single-channel SE model that aims to improve on temporally non-uniform noise degradation. It is based on MPSENet, using the same pre-processing, magnitude/phase decoders, and losses. It replaces the DenseNet encoder with a graph-based encoder (STGC) and the Conformer with BiMamba.

The encoder constructs a bipartite graph from frames with high estimated reliability (sources) to frames with low estimated reliability (targets). Edges are weighted by source reliability, cosine similarity between source and target frame, and a learned pairwise gate. Each target frame is updated as teh weighted sum of projected source frames.

Reliability is estimated per TF-bin using MCRA noise power estimation. From the noise power, the a priori SNR is estimated and used to compute a frame-level score. The paper also introduces Temporal Degradation Dispersion (TDD) as a proxy metric to quantify temporal non-uniformity of degradation. It is the coefficient of variation of time intervals between noise events.

Strengths

  • The model is well motivated, trying to leverage relatively clean speech segments in signals with non-stationary noise.
  • The reliability measure is based integrates signal processing principles in a DNN architecture and it is reference-free.
  • TDD enables ablations over different variations of temporal degradation, while keeping SNR constant.
  • The degradation of models across TDD levels is an interesting finding.
  • Reliability estimation is modular and can be built upon in future studies.
  • Graph well interpretable (reliability, frame similarity). Visualizations could be used to show information flow from source to target frames
  • The model performs competitively on the VB-DMD and DNS datasets. It is also least effected by non-uniform temporal degradation.

Weaknesses

  • Figure 2 and the text describe different architectures and some components are not specified.
  • Evaluation
    • The closest baseline (SEMamba) and more up-to-date SOTA models are missing.
    • Improvements are relatively small, no significance test.
    • Missing parameter counts, FLOPs, GMACs
  • Hyperparameters unspecified:
    • Eq. 3: power-law compression \(c\)
    • L156: number of Mamba blocks
    • Eq. 7: reliability threshold
    • Hidden size in SFG layers

L74 - L77 Unsupported claim that processing speech as temporal sequence weakens contribution of highly informative frame. Transformers use attention for non-local information transfer, Mamba uses time-step modulation to suppress integration of irrelevant input features.

L182 - It seems like the figure and text describe different versions of the model. - The LTF and STGC modules in Figure 2 are inconsistent with the progressive fusion approach in section 3.1.2. In the text, both modules are a single, interleaved process. The figure shows concatenation followed by convolutions. - The output \(T_{f}\) of the LTF module is not introduced in the text or figure. - The LTF and SFG modules both have F/2 in the last dimension. It is not specified how and where this downsampling happens. - The Bi-Mamba module in the figure uses two parallel forward/backward passes with RMSNorm in between. This is not the same as described in section 3.2. - It is unclear why RMSNorm is only used after the first Mamba block. - A second input to the plus symbol after the RMSNorms is missing.

L202 Reliability is computed from SNR (Eq. 5) based on MCRA estimates of the noise power. MCRA is using minimum statistics to track the noise floor. If a noise burst occurs in otherwise clean speech, the a posteriori SNR would be very high, leading to high reliability values for very noisy frames. So noise burst frames might be selected as reliable source frames.

L211-L215 The threshold \(\tau_{q}\) is never specified. Also frames that are neither in the top \(K\) or below the threshold are not part of the graph. It is unclear how they are updated (no updated happening at all?).

L222 Eq. 9 introduces acoustic similarity based on cosine similarity of input frames, which has several problems: - Each frame consists of concatenated compressed magnitude and phase (L197). The wrapped phase is highly noisy and sensitive to tiny time-shifts, making the measure highly noisy as well. - It treats a magnitude variations in low frequency bins the same as in high frequency bins, which does not fit logarithmic perception of humans. - The similarity is computed on noisy frames, so it also tracks noise similarity.

L236 Eq. 11 completely drops the degraded frame, building the target-frames in deeper layers of the SFG only from the source frames. These might not even be related to the speech content of the degraded frame.

L258 The distinction between LTF and STGC is unclear. Figure 2 suggests that SFG and LTF are two components where the extracted features are fused in the STGC module. Eq. 12 and 13 suggest that LTF and STGC are an interleaved sequential process.

L260 Introduces \(\mathcal{A}_{i}\) that aligns LTF and SFG output dimensions, but does not specify how this is done exactly (projection, down/upsampling, pooling, …?).

L295 The BiMamba input \(A_{in}\) drops the frequency dimension that is still present in the output of the STGC module. Its unclear whether this is a mistake, frequency is projected and squeezed, or frequency is folded into channel dimension.

L298 In Eq 14, \(H_{backward}\) is not re-reversed. In Eq 15, the forward and backward representations are concatenated along the channel dimensions. Without re-reversal, the frames are misaligned.

L304 The loss formulation does not fit the BiMamba section. If it should reflect the losses of MPSENet, then it is missing the time-loss. If not, the losses need clearer specification.

L325 - Table 1 lacks any parameter counts, FLOPs or GMACs, comparison of the models is thus very unclear. - It also lacks the obvious direct baseline SEMamba, which is performs on par with SFG-Mamba on VB-DMD and outperforms SFG-Mamba on DNS, without any graph modeling. - The conference-version of MPSENet was never trained on DNS, so the results here must be from retrained version, which is not stated. - The more up-to-date journal-version of MPSENet reports better results than SFG-Mamba. - The table lacks SOTA disciminative Mamba models like SEMamba, RWSA-MambaUNet, USEMamba, or similar. aTENNuate not really comparable as it is a low-latency causal raw-waveform SE model

L352 Its unclear why different metrics are used for the datasets, and why there are no non-intrusive metrics for DNS.

L358 Improvements lack any statistical significance tests. Compared to SEMamba, the only improvement are 0.02 PESQ (same COVL), so the actual advantage of SFG-Mamba is absolute performance is unclear without parameter counts.

L371-374 The differences mentioned here are not clearly noticeable. Also, a better harmonic structure does not relate to the postulated advantage of SFG-Mamba for non-uniform noise degradation.

L411-413 A dataset construction is described, that includes transient interference events and stationary background noise. It is not stated, which data is used here (real-world noise events, Gaussian noise, …).

L422 Unclear why TDD values would only span [0,1]. Concentrated noise events with clean speech segments between could yield higher TDD values (high std, low mean).

L439 Figure 5 introduces SSNR metric without any introduction.

L452 LTF component is part of the STGC fusion, so the "-STGC" ablation would remove the encoder entirely. What is the input the BiMamba then? Also, ablation dropping the reliability estimate would be informative about the efficacy of the proposed mechanism.

L457 It is explicitly mentioned that parameter counts are kept comparable, why not mention the parameter counts anywhere?

L466 The figure 5 does not support the claim of gradual decline for more connections. In fact, at \(K=30\), a second peak is observed, with even higher CBAK and SSNR. Also, metric variance would be useful as the metrics vary only in a very small band.

L480 VB-DMD and DNS might be too easy, observed differences are very small. More challenging current datasets would be informative.

Formulations

  • Citation-style seems to be broken
    • Lack of parentheses
    • L129 missing year
    • L122 & L123 duplicated citation
  • Mathematical notation
    • L46 uses \(\mathbf{d}\) instead of \(\Delta t\)
    • L239 introduces \(\mathcal{N}_\text{src}(j)=\mathcal{V}_{s}\) instead of just directly using \(\mathcal{V}_{s}\)
    • L412 introduces \(K\) as the number of transient noise events, colliding with \(K\) in Eq. 7 in a similar context.
    • The graph definition L215 is untypical for a weighted graph. Would expect vertices, edges and weights, i.e. \(G=(V, E, W)\)