吉林大学学报（理学版）Issue(2)：273-279,7.DOI:10.13413/j.cnki.jdxblxb.2015.02.23
支持大规模流数据处理的在线MapReduce 数据传输机制
Online MapReduce Data Transmission Mechanism Supporting Large-Scale Stream Data Processing
摘要
Abstract
We proposed a scalable and dynamic MapReduce computation model which supports the online processing of large-scale dynamic/static data against the characteristics of uneven stream data size and dynamic flowing and breaking out suddenly.On this basis,we proposed an online MapReduce data transmission mechanism and implemented its prototype program based on the push mode of Event and the use of Netty asynchronous communication technology.This paper focuses on solving fast online transfer of the large-scale distributed computing program and data dynamic distribution to provide support for dynamic MapReduce model.The experimental results show that the method can greatly improve the transmission efficiency of data between jobs compared with the traditional socket pipeline method in Hadoop system and improve real-time data stream handling significantly.关键词
大数据/流数据处理/MapReduce模型/数据传输机制Key words
big data/stream data processing/MapReduce model/data transmission mechanism分类
信息技术与安全科学引用本文复制引用
魏晓辉,李聪,李洪亮,李翔,刘圆圆,李丽娜,庄园..支持大规模流数据处理的在线MapReduce 数据传输机制[J].吉林大学学报（理学版）,2015,(2):273-279,7.基金项目
国家自然科学基金(批准号:61170004)、教育部高等学校博士学科点专项科研基金(批准号:20130061110052)、吉林省科技发展计划重点科技攻关项目(批准号:20140204013GX)、深部探测技术与实验研究专项基金(批准号:SinoProbe-09-01)和吉林大学基本科研业务费专项基金(批准号:450060491439) （批准号:61170004）
