中国光学（中英文）2023，Vol.16Issue(6)：1343-1355,13.DOI:10.37188/CO.2023-0036
基于异构光子神经网络的多模态特征融合
Multimodal feature fusion based on heterogeneous optical neural networks
摘要
Abstract
Current study on photonic neural networks mainly focuses on improving the performance of single-modal networks,while study on multimodal information processing is lacking.Compared with single-modal networks,multimodal learning utilizes complementary information between modalities.Therefore,multimodal learning can make the representation learned by the model more complete.In this paper,we pro-pose a method that combines photonic neural networks and multimodal fusion techniques.First,a heterogen-eous photonic neural network is constructed by combining a photonic convolutional neural network and a photonic artificial neural network,and multimodal data are processed by the heterogeneous photonic neural network.Second,the fusion performance is enhanced by introducing attention mechanism in the fusion stage.Ultimately,the accuracy of task classification is improved.In the MNIST dataset of handwritten digits classi-fication task,the classification accuracy of the heterogeneous photonic neural network fused by the splicing method is 95.75％;the heterogeneous photonic neural network fused by introducing the attention mechanism is classified with an accuracy of 98.31％,which is better than many current advanced single-modal photonic neural networks.Compared with the electronic heterogeneous neural network,the training speed of the mod-el is improved by 1.7 times;compared with the single-modality photonic neural network model,the hetero-geneous photonic neural network can make the representation learned by the model more complete,thus ef-fectively improving the classification accuracy of MNIST dataset of handwritten digits.关键词
光子神经网络/多模态/注意力机制Key words
photonic neural network/multimodal/attention mechanism分类
信息技术与安全科学引用本文复制引用
郑一臻,戴键,张天,徐坤..基于异构光子神经网络的多模态特征融合[J].中国光学（中英文）,2023,16(6):1343-1355,13.基金项目
国家自然科学基金资助(No.62171055,No.61705015,No.61625104,No.61821001,No.62135009,No.61971065) （No.62171055,No.61705015,No.61625104,No.61821001,No.62135009,No.61971065）
国家重点研发计划资助(No.2019YFB1803504) （No.2019YFB1803504）
信息光子学与光通信国家重点实验室(北京邮电大学)基金资助(No.IPOC2020ZT08,No.IPOC2020ZT03)Supported by the National Natural Science Foundation of China(No.62171055,No.61705015,No.61625104,No.61821001,No.62135009,No.61971065) （北京邮电大学）
National Key Research and Development Program(No.2019YFB1803504) （No.2019YFB1803504）
the State Key Laboratory of Information Photonics and Optical Communications(Beijing University of Posts and Telecommunications)(No.IPOC2020ZT08,No.IPOC2020ZT03) （Beijing University of Posts and Telecommunications）
