计算机工程与应用2019，Vol.55Issue(23)：64-69,6.DOI:10.3778/j.issn.1002-8331.1808-0342
一种高效的Flink与MongoDB连接中间件的研究与实现
Research and Implementation of Efficient Connection Middleware Between Flink and MongoDB
摘要
Abstract
In order to improve the reading and writing rate between big data processing platform Flink and MongoDB, this paper proposes and implements an efficient connection middleware of Flink and MongoDB. Based on Flink's paral-lelization idea, by logically fragmenting the data, the interface in the Mongo-Java package is called to realize parallel reading and writing of data. With different scale of hydrological sensor datasets as experimental data, the reading and writing speeds of the data in Java single-threaded operation, Hadoop and MongoDB connector and the Flink and MongoDB connection middleware proposed in this paper are tested. The results show that the efficiency of using Flink to read and write data is 1.5 times higher than the single-threaded operation, which validates that the connection middleware can effec-tively improve the reading and writing speed of massive data.关键词
Flink/MongoDB/Flink与MongoDB连接中间件/数据逻辑分片/并行读写Key words
Flink/MongoDB/Flink-MongoDB connector middleware/logically fragmenting data/parallel reading and writing分类
信息技术与安全科学引用本文复制引用
胡程,叶枫..一种高效的Flink与MongoDB连接中间件的研究与实现[J].计算机工程与应用,2019,55(23):64-69,6.基金项目
2017江苏省博士后科研资助计划(No.1701020C) （No.1701020C）
2017江苏省"六大人才高峰"资助项目(No.XYDXX-078) （No.XYDXX-078）
中央高校基本业务费项目(No.2013B01814). （No.2013B01814）
