现代信息科技2024，Vol.8Issue(6)：29-34,6.DOI:10.19850/j.cnki.2096-4706.2024.06.007
基于有状态实时流的流批一体数据处理平台的设计与研究
Design and Research of a Flow Batch Integration Data Processing Platform Based on Stateful Real-time Flow
摘要
Abstract
Today,the scale and complexity of data are constantly increasing,and the requirements for data processing platforms are also increasing.Traditional batch processing and real-time processing technologies have their own advantages and disadvantages,making it difficult to meet the needs of large-scale data processing.Therefore,a data processing platform that integrates flow processing and batch processing has emerged.On the basis of discussing the core architecture design of flow batch integration,this paper proposes a data processing method for flow batch integration based on stateful real-time flow,and implements the processing and calculation of flow batch integration data through a platform based approach.This platform has been demonstrated application in Sichuan Expressway Group and Guiyang government units.The application results show that the platform not only unifies batch processing and flow processing frameworks,but also has the advantages of efficiency,reliability,scalability,and can meet the needs of large-scale data processing.The implementation of this platform is of great significance for improving data processing efficiency and accuracy.关键词
批处理/有状态实时流/平台化/流批一体Key words
batch processing/stateful real-time flow/platformization/flow batch integration分类
信息技术与安全科学引用本文复制引用
周维,曹扬,谢红韬,胡建..基于有状态实时流的流批一体数据处理平台的设计与研究[J].现代信息科技,2024,8(6):29-34,6.基金项目
国家自然科学基金(U19B2027) （U19B2027）
