摘要
Abstract
With the widespread adoption of large vision-language models in multimodal tasks,their security faces severe challenges.Among them,jailbreak attacks bypass safety mechanisms through adversarial samples and carefully crafted prompts,inducing models to generate harmful content.Existing defense approaches mainly focus on enhancing the safety of model outputs or improving the accuracy of harmful sample detection.However,the former often reduces discriminability and increases computational costs,while the latter usually lacks post-alignment mechanisms and cannot effectively intervene in model outputs.To address these issues,this paper proposes a novel multimodal defense method based on steering vector detection and dual-alignment mechanism.Specifically,the method employs pre-generated steering vectors to detect harmful outputs,thereby improving both detection accuracy and efficiency.A subsequent alignment mechanism is applied to filter the outputs,ensuring response safety.To construct training data,this paper generates adversarial images corres-ponding to harmful queries and responses,paired with the same harmful queries and original images to form a new dataset.The data are fed into the model to extract hidden states under both harmful and benign inputs.By computing the average differences between positive and negative samples,this paper obtains steering vectors and thresholds for jailbreak detection.During inference,model hidden states are projected onto the steering vectors,and inputs are classified as harmful or benign based on the threshold.If deemed harmful,the defense mechanism is triggered,where an external model generates candidate outputs that are scored and filtered to produce safe and useful responses.Experimental results demonstrate that the proposed method achieves higher detection accuracy and robustness against harmful samples across multiple datasets,while signifi-cantly improving output safety.Meanwhile,sample detection is conducted without the involvement of external models,effectively reducing computational costs.关键词
引导向量/视觉语言模型/越狱攻击/多模态防御/隐藏状态Key words
steering vector/visual-language model/jailbreak attack/multimodal defense/hidden state分类
信息技术与安全科学
