电子学报2026，Vol.54Issue(3)：1105-1117,13.DOI:10.12263/DZXB.20251129
面向单目图像下三维目标检测与空间语义描述的统一建模
Towards Unified Modeling of 3D Detection and Spatial Semantic Captioning with Monocular Images
摘要
Abstract
Monocular 3D object detection provides a low-cost solution for 3D perception.However,existing methods struggle to generate scene descriptions that are intuitively understandable to humans,which limits their applicability in sce-narios requiring rich semantic understanding,such as human-computer interaction and autonomous driving.As a direct man-ifestation of human linguistic intelligence,visual captioning offers an ideal communication medium,endowing machines with the ability to intuitively"narrate"a scene.However,existing visual captioning methods primarily focus on monocular image content and can only describe two-dimensional topological relationships between objects,lacking the capability to ac-curately model and express 3D geometric information(e.g.,precise distance,spatial location,and motion state).If a two-stage approach is adopted—first performing 3D detection and then leveraging a large model to generate descriptions—it suf-fers from low system efficiency and poor information consistency.Moreover,the descriptions generated by large models are limited to topological relationships and fail to accurately reflect 3D geometric information.In addition,the inherent"halluci-nation"phenomenon of large models often leads to inaccurate spatial information accompanied by redundant descriptions.To address these issues,this paper proposes for the first time a novel task:monocular 3D detection and captioning(Mono3DDC).This task aims to unify monocular 3D object detection with caption generation,requiring the model to simul-taneously learn depth-aware visual features and linguistic semantics.Through an end-to-end network architecture,the gener-ated descriptions are enabled to accurately capture consistent 3D spatial information,ensuring both geometric accuracy in the descriptions and high precision in 3D object detection.To support in-depth exploration in this interdisciplinary research area,we construct the first benchmark dataset for monocular 3D visual captioning with Chinese semantic support,named KITTI3DDC.Based on the KITTI dataset,this dataset employs an efficient automated data generation pipeline that leverag-es the synergy between a large language model and structured verification templates.While ensuring diversity and linguistic fluency,the pipeline strictly maintains geometric consistency in spatial information,providing high-quality multimodal su-pervision signals for subsequent research.Furthermore,we design a novel unified framework,Mono3DDC-TR(Monocular 3D Detection and Captioning based on Transformer).By deeply integrating optimized geometric and visual features,this framework achieves significant advantages in both caption generation accuracy and multi-category 3D detection perfor-mance.It attains state-of-the-art results on the constructed KITTI3DDC benchmark,validating the effectiveness of the pro-posed end-to-end unified framework in jointly modeling geometric information and linguistic semantics.This paper pro-vides a comprehensive benchmark for the Mono3DDC task,effectively promoting its development and practical application.关键词
多模态学习/场景理解/三维目标检测/三维视觉描述/单目视觉/端到端学习Key words
multimodal learning/scene understanding/3D object detection/3D visual captioning/monocular vision/end-to-end learning分类
信息技术与安全科学引用本文复制引用
王天添,郭柯宇,罗函轲,孙士杰,程惠泽,孙张龙..面向单目图像下三维目标检测与空间语义描述的统一建模[J].电子学报,2026,54(3):1105-1117,13.基金项目
国家重点研发计划(No.2023YFB4301800) （No.2023YFB4301800）
国家自然科学基金(No.62576050) （No.62576050）
江西省青年基金(No.S2024QNJJL0062) National Key Research and Development Program of China(No.2023YFB4301800) （No.S2024QNJJL0062）
National Natural Science Foundation of China(No.62576050) （No.62576050）
The Jiangxi Province Science Foundation Fund(No.S2024QNJJL0062) （No.S2024QNJJL0062）
