仇瑜 1冯珺 2郑哲慧 3赵祎 4宋昊旻 2陈祖歌 2王绍兰3
作者信息
- 1. 智谱AI,北京 100084
- 2. 国网浙江省电力有限公司信息通信分公司,杭州 310020
- 3. 智谱AI,北京 100084||清华大学计算机科学与技术系,北京 100084
- 4. 清华大学计算机科学与技术系,北京 100084
- 折叠
摘要
Abstract
[Objective]Significant progress has been made in applying large language models(LLMs)to knowledge-based visual question answering(VQA),where systems jointly reason over visual content and external knowledge to produce accurate answers.However,existing approaches are limited in specialized vertical domains,particularly the power industry.A major challenge lies in framing effective prompts for LLMs.Given the scarcity of domain-specific textual corpora and the highly technical nature of power industry system operations,traditional prompt engineering methods often fail to provide sufficient contextual grounding.Consequently,even powerful general-purpose LLMs are unable to fully exploit their reasoning capabilities,resulting in suboptimal performance and limited practical utility.Moreover,most existing studies rely heavily on proprietary,closed-source models,such as GPT-4,for inference in VQA tasks.Despite these models' impressive zero-shot capabilities,their use incurs substantial computational costs,application programming interface latency,and a reliance on third-party services,hindering scalability,reproducibility,and real-world deployment,particularly in industrial settings that require data privacy,low-latency responses,and cost efficiency.These constraints underscore the need for an open,efficient,and domain-adapted alternative that can deliver high accuracy without sacrificing autonomy or affordability.[Methods]This paper proposes a novel large-scale model-based visual question-answering framework that is tailored to the power industry and centered on contextual knowledge prompting.This method leverages a foundational vision-language model that generates initial contextual knowledge examples from input image-question pairs.These examples encapsulate relevant visual semantics and preliminary reasoning traces.Subsequently,we introduce a lightweight answer selection layer that produces a set of plausible candidate answers from multimodal features.Crucially,the generated contextual knowledge examples and candidate answers are dynamically integrated into a structured prompt template,which is then fed to an LLM for final reasoning and answer refinement.This design effectively bridges the gap between generic visual understanding and domain-specific knowledge,enabling the LLM to"reason with context"rather than relying on its internal(and often incomplete)pre-trained knowledge.In alignment with our goals of accessibility and sustainability,we deliberately use LLaMA,an open-source,freely available LLM,as the backbone of our system,replacing expensive alternatives such as GPT-4.To further enhance domain adaptation,we curate a small but high-quality dataset comprising annotated image-question-answer triples from real-world power infrastructure scenarios(e.g.,substation equipment identification,fault diagnosis from thermal images,and safety compliance checks).This dataset is used for finetuning the LLaMA-based VQA pipeline using parameter-efficient techniques,such as low-rank adaptation,to achieve rapid adaptation with minimal computational overhead.[Results]We evaluate our proposed method on two established knowledge-intensive VQA benchmarks:EVQA and A-OKVQA.The experimental results demonstrate that our contextual knowledge-prompting strategy significantly outperforms state-of-the-art baselines,achieving absolute accuracy gains of 8.8％on EVQA and 14.5％on A-OKVQA,validating the efficacy of our prompt construction mechanism and the viability of open-source LLMs in specialized industrial applications.[Conclusions]This work advances the technical frontier of domain-specific VQA and provides a practical,cost-effective,and reproducible blueprint for deploying large-model intelligence in critical infrastructure sectors.关键词
电力领域/知识提示/大语言模型/视觉问答Key words
power domain/knowledge prompt/large language model/visual question answering分类
信息技术与安全科学
