Music source separation method based on time and channel dual-dimensional sequential attention
-
摘要:
针对音乐中乐器声源表征特异性不足的问题, 结合乐器声源与曲式内容相关的结构特征, 提出了双维序列注意力/时域端到端音乐源分离方法。首先, 由于各乐器声源在曲式不同部分的出现具有显著规律性, 因此从时间及特征通道两个维度, 对特征基函数进行差异化注意力加权。其次, 在损失函数中引入频率多分辨因子, 同时从时域及频域衡量分离后声源和理想声源间的差异。在MUSDB18数据集上的实验结果表明, 同时强调声源的时域曲式结构特征和离散谐波特征, 可以进一步改善乐器声源分离效果。与目前最先进的时域端到端音乐源分离方法Demucs相比, 信噪比指标提升了0.40 dB, 且在鼓和低音等声源的分离上表现尤为突出, 鼓声源信噪比指标提升0.13 dB, 低音声源信噪比指标提升0.60 dB。充分利用声源的语义内容及声学特征等多维度先验知识, 可以进一步提升声源表征的特异性, 从而提升声源的可分离程度。
Abstract:An end-to-end time-domain music source separation method based on dual-dimension sequential attention combing the structural characters related to instrument sound sources and contents of song pattern is proposed to address the insufficient specificity when characterizing the instrument sound sources in music. First, characteristic basis functions are weighted with different attention based on two dimensions, namely time and characteristic channels, because of the significant regularity of the occurrence of different instrument sound sources in different parts of the song pattern. Second, a multi-resolution frequency factor is introduced into the loss function to measure the difference between separated sound sources and ideal ones from both time and frequency domains at the same time. As shown by the experimental results on the MUSDB18 dataset, the separation results of instrumental sound sources can be improved by giving special attention to both the time-domain song pattern structure features and discrete harmonic features of the sound sources. Compared with Demucs, the most advanced end-to-end time-domain music source separation method, the signal-to-noise ratio index of this method is improved by 0.40 dB, with particularly outstanding performances in the separation of drum and bass audio sources, whose signal-to-noise ratio index is improved by 0.13 dB and 0.60 dB, respectively. The specificity of the characterization of sound sources can be improved through sufficient use of multi-dimensional
a priori knowledge, such as the semantic content and acoustic feature of sound sources, thus improving the degree of separability of sound sources. -
图 7 音乐片段“Al James”分离前后波形图对比 (a)分离后低音; (b)原始低音; (c)分离后鼓声; (d)原始鼓声; (e)分离后人声; (f)原始人声音乐; (g)分离后其他乐器; (h)原始其他乐器
图 8 音乐片段“Al James”分离前后语谱图对比 (a)分离后低音; (b)原始低音; (c)分离后鼓声; (d)原始鼓声; (e)分离后人声; (f)原始人声音乐; (g)分离后其他乐器; (h)原始其他乐器
表 1 DDSA-Unet网络参数
网络模块 网络层 网络参数 广义编码层ii = {1, 2, 3, 4, 5, 6} 一维卷积 [8,4,64×2i − 1] 组归一化 [32,64×2i − 1] 通道序列注意力块 [16,64×2i − 1] 一维卷积 [1,1,64×2i] 组归一化 [32,64×2i] 循环网络层 简单循环单元 64 广义解码层ii = {1, 2, 3, 4, 5, 6} 一维卷积 [1,1,64×2i] 组归一化 [32,64×2i] 一维卷积 [8,4,64×2i − 1] (i < 6), [8,4,8] (i = 6) 组归一化 [32,64×2i − 1] (i < 6), N/A (i = 6) 通道序列注意力块 [16,64×2i − 1] (i < 6), N/A (i = 6) 
下载: 导出CSV
表 2 双维度注意力块不同池化方式性能对比
网络模型 SDR (dB) SAR (dB) SIR (dB) DDSA-Unet (混合池化) 5.39 5.86 11.01 DDSA-Unet (最大池化) 5.26 5.53 10.65 DDSA-Unet (平均池化) 5.58 5.94 10.94
下载: 导出CSV
表 3 双维度注意力块不同排列方式不同池化方式模型性能对比
网络模型 SDR (dB) SAR (dB) SIR (dB) DDSA-Unet (并行, 平均池化) 5.47 5.88 10.41 DDSA-Unet (通道模块在前,平均池化) 5.58 5.94 10.94 DDSA-Unet (时间模块在前,平均池化) 5.34 5.81 10.18 DDSA-Unet (并行, 最大池化) 5.12 5.81 10.41 DDSA-Unet (通道模块在前, 最大化) 5.26 5.53 10.65 DDSA-Unet (时间模块在前, 最大池化) 4.98 5.65 10.21 DDSA-Unet (并行, 混合池化) 5.37 5.74 10.07 DDSA-Unet (通道模块在前, 混合池化) 5.39 5.86 10.91 DDSA-Unet (时间模块在前, 混合池化) 5.10 5.61 9.96
下载: 导出CSV
表 4 采用不同注意力块的模型性能对比
网络模型 批尺寸 训练速度(min) SDR(dB) SAR(dB) SIR(dB) DDSA-Unet 9 19.0 5.58 5.94 10.94 SE-Unet 9 17.9 5.38 5.95 10.87 SK-Unet 6 19.4 5.21 5.84 10.38
下载: 导出CSV
表 5 不同损失函数网络模型性能对比
损失函数范数形式 SDR (dB) SAR (dB) SIR (dB) 多分辨率STFT 5.74 6.02 11.07 L2 5.53 5.92 10.32 Smooth L1 5.41 5.87 10.77 L1 5.58 5.94 10.94 L1 (L2正则化) 5.48 5.91 10.48 L2 (L2正则化) 5.38 5.87 10.57
下载: 导出CSV
表 6 不同结构网络模型性能对比
网络模型 通道数 SDR (dB) SAR (dB) SIR (dB) Demucs 64 5.34 5.73 11.02 Demucs 100 5.58 6.08 10.39 DDSA-Unet (L1) 64 5.58 5.94 10.94 Demucs (多分辨率STFT) 64 5.55 5.91 10.74 DDSA-Unet (多分辨率STFT) 64 5.74 6.02 11.07
下载: 导出CSV
表 7 在MUSDB18数据集下不同模型的性能对比
模型 SDR (dB) SAR (dB) SIR (dB) 平均 鼓 低音 人声 其他 平均 平均 Wave-U-Net 3.23 4.22 3.21 2.25 3.25 4.49 6.26 Open-Unmix 5.33 5.73 5.23 6.32 4.02 5.90 10.49 Demucs(100通道) 5.58 6.08 5.83 6.29 4.12 6.08 10.39 DDSA-Unet 5.74 6.21 6.43 6.31 3.99 6.02 11.07
下载: 导出CSV
-
[1] 尹辉, 谢湘, 匡镜明. 基于听觉模型与自适应分数阶Fourier变换的声学特征在语音识别中的应用. 声学学报, 2012; 37(1): 97—103 doi: 10.15949/j.cnki.0371-0025.2012.01.011 [2] Zapata J R, Davies M, Gomez E. Multi-feature beat tracking. IEEE Trans. Audio Speech Lang. Process., 2014; 22(4): 816—825 doi: 10.1109/TASLP.2014.2305252 [3] Chang S, Lee K. Lyrics-to-audio alignment by unsupervised discovery of repetitive patterns in vowel acoustics. IEEE Access, 2017; 5: 16635—16648 doi: 10.1109/ACCESS.2017.2738558 [4] Rafii Z, Liutkus A, Stöter F, et al. An overview of lead and accompaniment separation in music. IEEE/ACM Trans. Audio Speech Lang. Process., 2018; 26(8): 1307—1335 doi: 10.1109/TASLP.2018.2825440 [5] Grais E M, Roma G, Simpson A J R, et al. Two-stage single-channel audio source separation using deep neural networks. IEEE/ACM Trans. Audio Speech Lang. Process., 2017; 25(9): 1773—1783 doi: 10.1109/TASLP.2017.2716443 [6] Stoller D, Ewert S, Dixon S. Wave-U-Net: A multi-scale neural network for end-to-end audio source separation. Proc. ISMIR, 2018 [7] Vincent E, Gribonval R, Févotte C. Performance measurement in blind audio source separation. IEEE Trans. Audio Speech Lang. Process., 2006; 14(4): 1462—1469 doi: 10.1109/TSA.2005.858005 [8] Weng W, Zhu X. INet: convolutional networks for biomedical image segmentation. IEEE Access, 2021; 9: 16591—16603 doi: 10.1109/ACCESS.2021.3053408 [9] 张天, 张天骐, 葛宛营, 等. 融合声源分离及反复结构模型的音乐分离方法. 声学学报, 2020; 45(5): 707—715 doi: 10.15949/j.cnki.0371-0025.2020.05.010 [10] Bellur A, Elhilali M. Bio-mimetic attentional feedback in music source separation. IEEE International Conference on Acoustics, Speech and Signal Processing, 2020: 8718—8722 [11] 汪斌, 陈宁. 基于残差注意力U-Net结构的端到端歌声分离模型. 华东理工大学学报(自然科学版), 2021; 47(5): 619—626 doi: 10.14135/j.cnki.1006-3080.20200903001 [12] 李相莲, 李明, 刘若伦, 等. 基于音色单元分布的音乐结构分析. 声学学报, 2010; 35(2): 276—281 doi: 10.15949/j.cnki.0371-0025.2010.02.028 [13] Perez-Lapillo J, Galkin O, Weyde T. Improving singing voice separation with the wave-u-net using minimum hyperspherical energy. IEEE International Conference on Acoustics, Speech and Signal Processing, 2020: 3272—3276 [14] Garoufis C, Zlatintsi A, Maragos P. HTMD-Net: A hybrid masking-denoising approach to time-domain monaural singing voice separation. 29th European Signal Processing Conference, IEEE, 2021: 341—345 [15] Défossez A, Usunier N, Bottou L, et al. Music source separation in the waveform domain. arXiv preprint: 1911.13254, 2019 [16] Yamamoto R, Song E, Kim J M. Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. IEEE International Conference on Acoustics, Speech and Signal Processing, 2020: 6199—6203 [17] Woo S, Park J, Lee J Y, et al. CBAM: Convolutional block attention module. European Conference on Computer Vision, 2018: 3—19 [18] Fu J, Liu J, Tian H, et al. Dual attention network for scene segmentation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3146—3154 [19] Rafii Z, Liutkus A, Stöter F R, et al. MUSDB18 − a corpus for music separation. 2017 [20] Uhlich S, Porcu M, Giron F, et al. Improving music source separation based on deep neural networks through data augmentation and network blending. IEEE International Conference on Acoustics, Speech and Signal Processing, 2017: 261—265 [21] Hu J, Shen L, Sun G. Squeeze-and-excitation networks. IEEE conference on computer vision and pattern recognition, 2018: 7132—7141 [22] Li X, Wang W, Hu X, et al. Selective kernel networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 510—519 [23] Stöter F R, Uhlich S, Liutkus A, et al. Open-Unmix − A reference implementation for music source separation. The Journal of Open Source Software, 2019; 4(41): 1667—1667 doi: 10.21105/joss.01667 [24] Takahashi N, Goswami N, Mitsufuji Y. MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation. 16th International Workshop on Acoustic Signal Enhancement, IEEE, 2018: 106—110 -
下载:
计量
- 文章访问数: 33
- 被引次数: 0
