南京农业大学学报2024，Vol.47Issue(4)：782-791,10.DOI:10.7685/jnau.202308009
基于多模态预训练模型的水稻病虫害图像描述生成研究
Research on image description generation of rice diseases and pests based on multimodal pre-training model
摘要
Abstract
[Objectives]Aiming at the lack of disease description in rice diseases and pests image classification technology,a lightweight rice diseases and pests image description model was proposed in this paper to describe rice diseases and pests image more specifically.[Methods]Ten common rice pests and diseases,such as rice bacterial blight,rice bacterial streak disease,rice bakanae disease,rice three chemical borers,rice blast,rice false smut,rice sheath blight,rice planthopper,rice thrip and rice brown spot were studied,and Chinese description data set of rice pests and diseases image was constructed.Firstly,the multimodal pre-training model CLIP was used to generate image vectors,which contained basic image information and rich semantic information.The mapping network was used to map the image vectors into the text space to generate text prompt vectors.Finally,the language model GPT-2 generates image descriptions according to the prompt vectors.[Results]The test results showed that the indexes of the model in this paper were significantly superior to other models in the image description data set of rice pests and diseases.Compared with the traditional CNN_LSTM model,the indexes of BLEU-1,BLEU-2,BLEU-3,BLEU-4,ROUGE and METEOR improved 0.26,0.27,0.24,0.22,0.22 and 0.14,respectively.And the generated image description had the advantages of accurate,detailed and rich semantics.The model was tested by using actual rice field pictures.The actual field scenes were more complex and diverse,and the generated image description index only slightly decreased compared with the data set index,which was still higher than other comparison models.The overall recognition accuracy of the model was 97.28%.[Conclusions]The image description method of rice diseases and pests based on multimodal pre-training model can accurately describe the rice diseases and pests,and provide a new idea for the detection of rice diseases and pests.关键词
多模态预训练模型/水稻病虫害/图像描述生成/诊断Key words
multimodal pre-training model/rice diseases and pests/image description generation/diagnosis分类
信息技术与安全科学引用本文复制引用
薛悦平,胡彦蓉,刘洪久,童莉珍,葛万钊..基于多模态预训练模型的水稻病虫害图像描述生成研究[J].南京农业大学学报,2024,47(4):782-791,10.基金项目
教育部人文社会科学研究规划基金项目(18YJA630037,21YJA630054) （18YJA630037,21YJA630054）
浙江省自然科学基金资助项目(LY18G010005) （LY18G010005）
