基于多模态数据的人体行为识别方法研究综述OA北大核心CSTPCD

Human action recognition(HAR)is widely applied in the fields of intelligent security,autonomous driving and human-computer interaction.With advances in capture equipment and sensor technology,the data that can be acquired for HAR is no longer limited to RGB data,but also multimodal data such as depth,skeleton,and infrared data.Feature extrac-tion methods in HAR based on RGB and skeleton data modalities are introduced in detail,including handcrafted-based and deep learning-based methods.For RGB data modalities,feature extraction algorithms based on two-stream convolu-tional neural network(2s-CNN),3D convolutional neural network(3DCNN)and hybrid network are analyzed.For skele-ton data modalities,some popular pose estimation algorithms for single and multi-person are firstly introduced.The classi-fication algorithms based on convolutional neural network(CNN),recurrent neural network(RNN),and graph convolu-tional neural network(GCN)are analyzed stressfully.A further comprehensive demonstration of the common datasets for both data modalities is presented.In addition,the current challenges are explored based on the corresponding data struc-ture features of RGB and skeleton.Finally,future research directions for deep learning-based HAR methods are discussed.