计算机技术与发展2024，Vol.34Issue(10)：118-125,8.DOI:10.20165/j.cnki.ISSN1673-629X.2024.0187
基于LSTM和位置增强的软提示向量优化
Optimization of Soft Prompt Vectors Based on LSTM and Position Enhancement
摘要
Abstract
Soft prompt learning is an emerging method for applying pretrained language models.However,the vectors generated by soft prompt learning may lack sequential structure,affecting the model's ability to define information at specific positions,resulting in impaired model performance.To address this,we delve into the sequential structure of soft prompt vectors and their influence on model performance.It was found that soft prompt vectors exhibit sequence sensitivity issues across different types of language models,model sizes,types of downstream tasks,and prompt lengths.In response,we propose a soft prompt sorting network based on LSTM and position enhancement.Firstly,an improved LSTM network is used for soft prompt sorting optimization,where a prompt selection gate is added at each gate to capture sequence information and generate well-ordered soft prompt vectors.Secondly,a position enhancement module is proposed for the sorting process,optimizing the order by combining absolute and relative position information.Tests on the GLUE dataset show that the proposed method brings an average performance improvement of 3.1%compared to baseline.关键词
软提示向量/序列结构/顺序敏感性/位置编码/长短期记忆Key words
soft prompt vector/sequential structure/order sensitivity/position encoding/long short-term memory分类
信息技术与安全科学引用本文复制引用
刘振东,程春玲,刘倩..基于LSTM和位置增强的软提示向量优化[J].计算机技术与发展,2024,34(10):118-125,8.基金项目
国家自然科学基金项目(61972201) （61972201）
