西安电子科技大学学报（自然科学版）2019，Vol.46Issue(1)：143-150,8.DOI:10.19665/j.issn1001-2400.2019.01.023
融合语音信号和脑电信号的多模态情感识别
Multimodal emotion recognition for the fusion of speech and EEG signals
摘要
Abstract
To construct an effective emotion recognition system,the emotions of joy,sadness,anger and neutrality are induced by sound stimulation,and the corresponding speech and EEG signals are collected. First,this paper extracts the nonlinear geometric feature and nonlinear attribute feature of EEG and speech signals by phase space reconstruction respectively,and the emotion recognition is realized by combining the basic features.Then,a feature fusion algorithm based on the Restricted Boltzmann Machine is constructed to realize multimodal emotion recognition from the perspective of feature fusion.Finally,a multimodal emotion recognition system is constructed through decision fusion by using the quadratic decision algorithm. The results show that the overall recognition rate of the multimodal emotion recognition system constructed by feature fusion is 1.08％ and 2.75％ higher than that of speech signals and that of EEG signals respectively,and that the overall recognition rate of the multimodal emotion recognition system constructed by decision fusion is 6.52％ and 8.19％ higher than that of speech signals and that of EEG signals respectively.The overall recognition effect of the multimodal emotion recognition system based on decision fusion is better than that of feature fusion.A more effective emotion recognition system can be constructed by combining the emotional data of different channels such as speech signals and EEG signals.关键词
语音信号/脑电信号/特征融合/决策融合Key words
speech signals/electroencephalo-graph signals/feature fusion/decision fusion分类
信息技术与安全科学引用本文复制引用
马江河,孙颖,张雪英..融合语音信号和脑电信号的多模态情感识别[J].西安电子科技大学学报（自然科学版）,2019,46(1):143-150,8.基金项目
国家自然科学基金(61371193) （61371193）
山西省青年科技研究基金(2013021016-2) （2013021016-2）
