西安电子科技大学学报（自然科学版）2024，Vol.51Issue(3)：136-146,11.DOI:10.19665/j.issn1001-2400.20231004
基于多注意力机制的纹理感知视频修复方法
Texture-aware video inpainting algorithm based on the multi-attention mechanism
摘要
Abstract
Existing video inpainting methods cannot effectively utilize distant spatial contents,which results in unreasonable structures and textures.To solve this problem,a texture-aware video inpainting algorithm based on the multi-attention mechanism is proposed in this paper.The algorithm designs a multi-attention mechanism composed of multi-head spatiotemporal attention and single-image local attention,guaranteeing global structures and enriching local textures.Multi-head spatial-temporal attention focuses on the overall spatial-temporal information,and single-image local attention distills local information through local windows of the self-attention mechanism.A plug-and-play fast Fourier convolution layer residual block is used to replace vanilla convolution in feedforward networks,expanding the receptive field into the entire image so that the global structure and texture of a single frame image can be enriched.The fast Fourier convolutional layer residual block and the single-image local attention complement each other and jointly promote the quality of local textures.Experimental results on YouTube-VOS and DAVIS datasets show that although the proposed method ranks second only to the optimal method Fuseformer on objective metrics,the number of parameters and running time are reduced by 54.8%and 21.5%respectively.And the proposed method can generate more visually realistic and semantically reasonable contents.关键词
视频修复/Transformer/快速傅里叶卷积/多注意力机制/纹理感知Key words
video inpainting/Transformer/fast Fourier convolution/multi-attention mechanism/texture-aware分类
信息技术与安全科学引用本文复制引用
夏译蓝,王秀美,程培涛..基于多注意力机制的纹理感知视频修复方法[J].西安电子科技大学学报（自然科学版）,2024,51(3):136-146,11.基金项目
国家自然科学基金(62372355,61972305,61871308) （62372355,61972305,61871308）
陕西省自然科学基础研究计划(2023-JC-ZD-39) （2023-JC-ZD-39）
陕西省重点研发计划(2021 ZDLGY02-03) （2021 ZDLGY02-03）
